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Abstract 


In  this  era  of  decreased  defense  budgets  and  limited  resources,  it  is  important  for 
decision-makers  to  determine  the  optimal  strategy  for  assessing  the  vulnerability  or  lethality  (V/L) 
of  a  weapon  system  and  the  role  of  Full-Up  System-Level  Live-Fire  Test  and  Evaluation  (FU  SL 
LFT&E)  in  that  strategy.  This  report  presents  a  foundation  for  a  methodology  to  (1)  identify, 
measure,  and  categorize  the  costs  and  benefits  of  FU  SL  LFT&E;  (2)  determine  the  relative 
significance  of  FU  SL  LFT&E  to  the  V/L  assessment  plan  of  the  weapon  system;  and  (3)  compare 
competing  V/L  assessment  plans  for  a  system. 

Descriptions  of  the  activities  of  FU  SL  LFT&E,  the  costs  and  benefits  (i.e.,  impacts)  of  those 
activities,  the  complexities  encountered  in  the  reporting  of  the  impacts,  and  approaches  for 
addressing  the  complexities  are  presented.  A  discussion  of  the  contributions  of  FU  SL  LFT&E 
to  a  V/L  assessment  strategy  and  suggestions  for  improving  the  cost-effectiveness  of  FU  SL 
LFT&E  are  included. 

The  Taxonomy  of  the  V/L  Analysis  Process  (Deitz  and  Ozolins  1989)  is  proposed  as  a 
framework  for  identifying  the  data  voids  to  be  addressed  in  a  V/L  assessment  plan.  The  potential 
of  adapting  the  principles  of  the  Cost  as  an  Independent  Variable  methodology  to  the  evaluation 
of  competing  plans  of  V/L  assessment  is  explored. 
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1.  Introduction 


U.S.  congressional  legislation  requires  weapon  systems  in  certain  categories  to 
undergo  Full-Up  System-Level  Live-Fire  Testing  and  Evaluation  (FU  SL  LFT&E)  prior 
to  the  system  entering  full-scale  production,  unless  a  waiver  is  requested  by  the  Service 
acquiring  the  system  and  granted  by  the  Secretary  of  Defense  [1].  The  waiver  request 
must  show  why  FU  SL  LFT&E  is  unreasonably  expensive  and  impractical  and  include 
an  alternative  strategy  for  assessing  the  survivability /lethality  of  the  system. 

Significant  resources  are  consumed  in  the  planning,  execution,  and  evaluation  of  a 
Full-Up  System-Level  Live-Fire  Test  (FU  SL  LFT).  In  this  era  of  decreased  defense 
budgets  and  limited  resources,  it  is  important  for  decision-makers  to  determine  the 
optimal  strategy  for  assessing  the  vulnerability  or  lethality  (V/L)  of  a  weapon  system 
and  the  role  of  FU  SL  LFT&E  in  that  strategy/  The  selection  of  the  optimal  assessment 
strategy  requires  that  a  consistent  methodology  be  in  place  for  the  identification  and 
measurement  of  the  costs  and  benefits  of  potential  assessment  plans,  the  weighing  of 
the  costs  against  the  benefits  for  plans  considered,  and  the  comparison  of  alternative 
competing  plans/ 


*  U.S.  Code  requires  Full-Up  System-Level  Live-Fire  Test  and  Evaluation  to  be  conducted  on  a  covered 
system  (any  vehicle,  weapon  platform,  or  conventional  weapon  system  that  includes  features  designed 
to  provide  some  degree  of  protection  to  users  in  combat  and  that  is  an  Acquisition  Category  I  or  II 
program),  a  major  munitions  program,  a  product  improvement  to  a  covered  system  or  a  major 
munitions  program,  or  a  missile  program  before  it  can  proceed  beyond  low-rate  initial  production  [1]. 

The  Secretary  of  Defense  may  waive  the  application  of  tests,  if  the  Secretary,  before  the  system  enters 
engineering  and  manufacturing  development  (EMD),  certifies  to  Congress  that  a  FU  SL  LFT  of  such 
system  or  program  would  be  unreasonably  expensive  and  impractical.  A  waiver  and  alternative 
LFT&E  plan  must  have  been  submitted  and  approved  by  Milestone  II  (i.e.,  approval  to  enter  EMD). 

*  FU  SL  LFT&E,  as  described  in  this  report,  addresses  questions  of  vulnerability  and  lethality,  not  the 
broader  concepts  of  system  survivability,  effectiveness,  and  susceptibility. 

*  In  Live-Fire  Test  (LFT)  legislation,  an  alternative  strategy  is  a  strategy  that  does  NOT  include  FU  SL 
LFT&E.  In  this  report,  however,  the  term  alternative,  as  in  alternative  strategy  or  alternative  plan,  is 
defined  more  broadly  and  is  used  to  indicate  more  than  one  option  or  a  choice  among  options. 
Therefore,  one  or  more  alternative  vulnerability  or  lethality  (V/L)  assessment  plans  or  strategies  may 
include  FU  SL  LFT&E  as  an  element. 
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The  ultimate  goal  of  the  Live-Fire  Test  and  Evaluation  (LFT&E)  process,  which 
includes  analysis,  experimental,  and  testing  activities  in  addition  to  FU  SL  LFT,  is  to 
quantify  the  V/L  characteristics  of  the  platform  or  munition.  Each  activity  of  the 
LFT&E  process  may  provide  an  important  element(s)  to  the  V/L  assessment  of  the 
system,  and  some  data  critical  to  the  assessment  may  be  provided  by  alternative 
activities.  When  designing  strategies  for  V/L  assessment,  it  is  therefore  important  to 
consider  the  whole  acquisition  cycle  and  all  potential  sources  for  the  data  needed 
by  evaluators  in  the  assessment  process. 

1.1  Research  Objectives.  This  study  seeks  to  establish  a  foundation  for  the 
development  of  a  methodology  to  (1)  identify,  measure,  and  categorize  the  costs  and 
benefits  of  FU  SL  LFT&E,  (2)  determine  the  relative  significance  of  FU  SL  LFT&E  (one 
element  of  a  V/L  assessment  plan)  to  the  V/L  assessment  plan  taken  as  a  whole,  and 
(3)  compare  competing  V/L  assessment  plans  for  a  system.  The  comparison  of 
assessment  plans  should  consider  the  costs,  benefits,  and  uncertainties  associated  with 
each  of  the  alternative  plans  presented. 

It  is  proposed  that  the  development  of  the  aforementioned  methodology  will 
provide  information  needed  for 

•  Ranking  alternative  plans  of  V/L  assessment. 

•  Budgeting  resources  for  the  FU  SL  LFT&E  element  of  a  V/L  assessment  plan. 

•  Making  specific  decisions,  including  the  appropriateness  of  a  FU  SL  LFT&E 
waiver  for  systems  covered  by  Live-Fire  Test  (LFT)  congressional  legislation. 

1.2  Research  Approach. 

1.2.1  Systems.  The  FU  SL  LFT&E  programs  of  three  U.S.  Army  weapon  systems 
included  under  the  LFT  congressional  mandate  are  analyzed  in  this  study: 
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•  Abrams  M1A2:  FU  SL  LFT&E,  an  element  of  the  vulnerability  assessment  plan  of 
the  weapon  system,  completed  in  1993. 

•  Army  Tactical  Missile  System  Block  IA  (ATACMS  Block  IA):  FU  SL  LFT&E,  an 
element  of  the  lethality  assessment  plan  of  the  weapon  system,  completed  in 
1996. 

•  Bradley  Fighting  Vehicle  System-A3,  M2A3/M3A3  (Bradley  A3)  FU  SL  LFT&E, 
an  element  of  the  vulnerability  assessment  plan  of  the  weapon  system,  scheduled 
to  be  completed  in  1999. 

1.2.2  Impacts  ofFU  SL  LFT&E.  The  specific  activities  that  define  the  FU  SL  LFT&E 
element  of  a  system's  assessment  strategy  are  identified.  For  each  system,  the 
components  of  each  identified  activity  and  the  costs  related  to  those  components  are 
determined.  In  addition,  the  following  practices  and  procedures  are  examined: 

•  Current  procedures  used  in  the  measurement  of  the  costs  related  to  identified 
activities. 

•  Current  practices  used  in  reporting  the  computed  costs. 

•  Current  practices  used  in  budgeting  for  the  costs  of  FU  SL  LFT&E  and 
computing  and  reporting  the  variances  between  actual  and  budgeted  costs. 

Identifying  the  benefits  of  FU  SL  LFT&E,  as  well  as  the  benefits  of  other  elements  of 
alternative  assessment  plans,  is  an  important  part  of  the  methodology  that  compares 
competing  assessment  strategies.  Although  there  is  a  large  element  of  subjectivity  in 
describing  these  benefits,  several  broad  categories  of  FU  SL  LFT  benefits  are  identified 
through  discussions  with  individuals  associated  with  the  V/L  assessment  plans  of  the 
systems  studied.  The  capabilities  of  an  element  of  a  V/L  assessment  plan  to  address  the 
critical  evaluation  issues  of  the  system  are  considered  to  be  the  benefits  of  that  plan 
element. 
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Complexities  encountered  in  identifying,  measuring,  and  reporting  the  costs  and 
benefits  or  impacts  (i.e.,  costs  and  benefits  are  often  described  by  the  term  impacts  in  the 
cost-benefit  literature)  of  FU  SL  LFT&E  and  its  associated  activities  are  recognized,  and 
approaches  for  addressing  these  complexities  are  suggested. 

It  is  proposed  that  the  methodology  used  in  identifying  and  measuring  the  impacts 
of  FU  SL  LFT&E  is  applicable  equally  to  the  computing  of  the  costs  and  benefits  of  other 
elements  of  the  assessment  strategies  for  weapon  systems. 

1.2.3  Comparison  of  Competing  Strategies.  The  Taxonomy  of  the  V/L  Analysis 
Process  (V/L  Taxonomy)*  provides  the  framework  for  identifying  (1)  the  specific  data 
set  required  for  an  adequate  assessment  of  the  critical  V/L  issues  of  a  weapon  system, 

(2)  the  subset  of  the  required  data  set  available  from  existing  reliable  sources,  and 

(3)  data  voids  to  be  addressed  in  a  V/L  assessment  plan. 

The  potential  of  adapting  the  principles  of  the  Cost  as  an  Independent  Variable 
(CAIV)  methodology  to  the  evaluation  of  competing  strategies  of  V/L  assessment  is 
explored.  The  analyses  of  the  identified  impacts  of  alternative  assessment  strategies, 
the  uncertainties  (i.e.,  risks)  associated  with  these  impacts,  and  the  priorities  assigned 
by  analysts  to  the  critical  data  voids  expected  to  be  produced  by  these  assessment 
strategies  serve  as  inputs  to  the  evaluation  approach  grounded  in  CATV  principles 
proposed  in  this  report. 

Complexities  encountered  in  predicting  the  data  or  information  to  be  obtained  from 
a  V/L  assessment  plan  prior  to  completing  the  plan  and  prioritizing  the  information 
needs  of  decision-makers  are  discussed. 

1.3  Organization  of  Report.  The  remainder  of  the  report  is  organized  as  follows. 
The  research  methodology  employed  in  this  study  is  developed  in  section  2  and 
includes  a  description  of  the  three  weapon  systems  studied  and  the  identification  of  the 


*  The  Taxonomy  of  the  V/L  Analysis  Process  was  first  introduced  in  Deitz  and  Ozolins  [2]. 
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agencies  that  provided  much  of  the  information  upon  which  this  report  is  based.  The 
results  of  the  research,  including  descriptions  of  the  activities  of  FU  SL  LFT&E,  the 
impacts  of  those  activities,  the  complexities  encountered  in  the  reporting  of  the  impacts, 
and  approaches  for  addressing  the  complexities  are  reported  in  section  3.  A  discussion 
of  Project  Manager  (PM)  personnel's  perceptions  of  the  role  of  FU  SL  LFT&E  in  the 
survivability  /lethality  assessment  process  and  their  suggestions  for  improving  the  cost- 
effectiveness  of  the  assessment  process  complete  this  section  of  the  report.  Section  4 
includes  a  discussion  of  the  application  of  the  V/L  Taxonomy  framework  and  CAIV 
principles  to  the  decision-making  process  of  selecting  the  optimal  V/L  assessment 
strategy.  The  report  concludes  in  section  5  with  a  discussion  of  the  relationship  of  FU 
SL  LFT&E,  cost-effective  assessment  strategies,  and  the  acquisition  process. 


2.  Research  Methodology 

2.1  Systems.  A  description  of  the  three  weapon  systems  examined  in  this  study 
and  a  brief  review  of  the  FU  SL  LFT&E  events  related  to  those  systems  are  provided  in 
the  following  paragraphs. 


2.1.1  Abrams  M1A2.  The  M1A2  Abrams  Main  Battle  Tank  system  is  described  as 
follows  in  the  "FY 1996  Annual  Report  of  the  Director,  Operational  Test  and 
Evaluation  [3]:" 

The  mission  of  the  Ml  A2  Abrams  tank  is  to  close  with  and  destroy  enemy 
forces  using  firepower,  maneuver,  and  shock  effect.  The  M1A2  is  an 
upgrade  of  the  Abrams  M1A1  intended  to  improve  target  acquisition  and 
engagement  rates,  improve  survivability,  and  maintain  operational 
suitability  at  the  level  of  the  M1A1.  Many  of  the  enhancements  were  made 
by  replacing  or  modifying  previous  items  to  take  advantage  of  the 
introduction  of  digital  distributed  data  and  power  architecture  of  the 
Ml A2.  The  Inter-Vehicular  Information  System  and  the  Position/ 

Navigation  equipment  are  designed  to  improve  battlefield  command, 
control,  and  communications.  The  Commander's  Independent  Thermal 
Viewer  increases  the  rate  of  target  acquisition  so  that  the  gunner  is  able  to 
engage  targets  more  rapidly. 


The  FU  SL  LFT,  an  element  of  the  vulnerability  assessment  plan  of  the  Abrams 
M1A2,  was  conducted  from  April  to  October  1993  at  the  U.S.  Army  Aberdeen  Test 
Center  (ATC)  at  Aberdeen  Proving  Ground  (APG),  MD.  A  total  of  11  shots  (i.ev 
10  shots  planned,  1  shot  repeated)  were  fired  against  2  of  the  initial  62  M1A2  vehicles 
produced  in  the  Low-Rate  Initial  Production  (LRIP)  contract.  The  Office  of  the 
Secretary  of  Defense  (OSD)  submitted  the  Abrams  M1A2  Live-Fire  Test  &  Evaluation 
Report  to  the  U.S.  Congress  in  May  1994. 


The  Abrams  Ml  A2  FU  SL  LFT  was  designed  to  investigate  the  effects  of  non¬ 
perforating  and  perforating  threats  likely  to  be  encountered  in  combat.  The  M1A2 
Abrams  Live  Fire  Vulnerability  Test  objectives  include  [4]: 


•  To  determine  the  vulnerability  of  the  M1A2  vehicle  and  total  system 
(crew  and  vehicle)  along  with  the  corresponding  loss  of  mobility  and/or 
firepower  function  as  a  result  of  nonperforating  or  perforating  threats 
likely  to  be  encountered. 

•  To  determine  the  vulnerability  of  the  crewmen  in  the  modified  crew 
compartment. 

•  To  investigate  the  survivability  improvements  incorporated  into  the 
Ml  A2  as  a  result  of  previous  testing. 

•  To  determine  if  there  were  unexpected  or  unacceptable  M1A2 
vulnerabilities  and  determine  how  these  vulnerabilities  might  be 
reduced  or  eliminated  if  they  were  significant. 

•  To  assess  the  repairability  of  the  vehicle. 

2.1.2  ATACMS  Block  IA.  The  ATACMS  Block  IA  system  is  described  as  follows  in 
the  "FY 1997  Annual  Report  of  the  Director,  Operational  Test  and  Evaluation  [5]:" 


The  Army  Tactical  Missile  System  (Army  TACMS)  is  a  family  of  long- 
range,  near  all-weather  guided  missiles  fired  from  the  Multiple  Launch 
Rocket  System  (MLRS)  M270  launcher  and  deployed  within  the 
ammunition  loads  of  corps  MLRS  battalions.  ...  The  Block  IA  is  an  upgrade 
intended  to  double  the  range  of  the  current  Army  TACMS  Block  I  missile. 


*  The  Aberdeen  Test  Center  (ATC)  was  formerly  known  as  the  Combat  Systems  Test  Activity  (CSTA). 
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Army  TACMS  Block  IA  will  dispense  M74  Anti-Personnel,  Anti-Materiel 
(APAM)  bomblets,  as  does  the  Block  I. . . . 

The  primary  changes  in  Army  TACMS  Block  IA  are  a  reduced  payload 
(from  950  to  300  bomblets)  and  the  addition  of  a  Global  Positioning  System 
(GPS)  navigation  aid  to  the  inertial  guidance  system.  The  reduced  payload 
is  necessary  to  achieve  the  extended  range.  To  compensate  for  the  reduced 
payload,  the  accuracy  of  the  missile  has  been  improved  with  inflight  GPS 
updates.  If  GPS  is  rendered  inoperable,  the  Army  TACMS  Block  IA  reverts 
to  inertial  guidance  only  and  maintains  Block  I  accuracy. 

The  FU  SL  LFT&E  of  the  ATACMS  Block  IA  did  not  include  vulnerability  tests  of 
the  chassis  and  launcher  of  the  missiles.  The  evaluation  of  the  FU  SL  LFT&E  of  the 
ATACMS  Block  IA  includes  the  results  of  3  test  events:  (1)  the  full-up  live-fire  arena 
tests,  conducted  from  May  to  June  1996  and  designed  to  observe  effects  of  23  M74 
bomblets  detonated  at  various  ranges  against  3  operationally  representative  threat 
targets,  (2)  the  2  developmental  and  operational  test  live-fire  launcher-to-target  firings 
(previously  referred  to  as  end-to-end  firings),  conducted  in  August  and  September  1996 
against  an  array  of  9  and  15  nonoperating  targets  and  designed  to  provide  a 
demonstration  of  ATACMS  potential  to  inflict  damage  upon  a  multiple-vehicle  area 
target,  and  (3)  the  tire  fragment  tests,  conducted  from  June  1994  to  June  1995  and 
designed  to  test  the  capability  of  M74  and  other  munitions  with  small  fragments  to 
inflict  damage  on  tires  (the  tire  tests  were  not  an  official  part  of  the  LFT  plan,  but  their 
results  were  included  in  LFT  evaluation).  The  arena  and  tire  fragment  tests  were 
conducted  respectively  at  ATC  and  the  Survivability /Lethality  Analysis  Directorate, 
U.S.  Army  Research  Laboratory  (SLAD/  ARL)  at  APG,  and  the  launcher-to-target  flight 
tests  were  performed  at  White  Sands  Missile  Range  (WSMR),  NM.  Milestone  in  for 
ATACMS  Block  IA  was  deferred  to  allow  more  time  to  address  issues  related  to 
acquisition,  in-flight  survivability,  and  lethality.  Follow-on  tests  were  conducted  both 
at  SLAD  and  ATC,  and  the  ATACMS  Block  IA  LFT&E  report  was  submitted  by  OSD  to 
the  U.S.  Congress  in  April  1998. 

The  ATACMS  Block  IA  FU  SL  LFT  was  designed  to  provide  information  to  assess 
the  ability  of  the  system  to  eliminate  and/or  degrade  the  operational  functions  of 
representative  threat-target  systems,  thus  providing  insights  into  the  principal  damage 
mechanisms  and  target  failure  modes  occurring  as  a  result  of  munitions  and  target 
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interaction.  The  "Live  Fire  Independent  Evaluation  Plan/Test  Design  Plan  (IEP/TDP) 
for  the  Army  Tactical  Missile  System  (ATACMS)  Block  IA"  reports  the  critical  test  and 
evaluation  issues  of  the  ATACMS  Block  IA  program  as  follows  [6]: 


•  Bomblet  lethality  . . .  What  is  the  lethality  of  the  M74  bomblet  against 
personnel  and  light  materiel  targets?  Subissues  include:  What  are  the 
M74  fragmentation  characteristics?  What  are  the  penetration 
characteristics  of  the  M74  fragments?  Can  the  M74  fragments  routinely 
damage  components  of  the  desired  targets?  How  is  the  performance  of 
the  M74  bomblet  affected  by  degraded  conditions  such  as  uneven 
terrain  and  various  impact  media? 

•  Pattern  lethality  . . .  What  is  the  lethality  of  the  ATACMS  Block  IA  missile 
against  the  required  target  elements,  given  the  element  is  in  the  bomblet 
pattern?  Subissues  include:  What  is  the  density  and  distribution  of  the 
M74  bomblets  within  the  bomblet  ground  pattern?  What  is  the  M74 
bomblet  dud  rate  and  distribution  within  the  ground  pattern? 

•  System  Effectiveness  . . .  What  is  the  system  effectiveness  of  the  ATACMS 
Block  IA  system  against  the  required  target  sets  to  include  both  point 
and  area  targets?  Subissues  include:  What  is  the  system  accuracy 
relative  to  the  aimpoint?  What  are  the  pattern  dimensions?  What  are 
the  target  dimensions  (area  targets)?  Vttiat  is  the  Target  Location  Error 
for  the  target  of  interest? 

2.1.3  Bradley  A3.  The  Bradley  A3  system  is  described  as  follows  in  the  "FY 1998 
Annual  Report  of  the  Director,  Operational  Test  and  Evaluation  [7]:" 

The  M2A3  and  M3A3  Bradley  Fighting  Vehicle  Systems  (BFVS)  are 
improved  versions  of  the  M2A2  and  M3A2  BFVS.  The  BFVS- A3  includes 
enhancements  intended  to  improve  lethality,  mobility,  survivability,  and 
sustainability.  Additionally,  these  enhancements  are  intended  to  provide 
increased  situational  awareness  and  digital  command  and  control 
capabilities  necessary  to  provide  information  superiority  to  the  dominant 
maneuver  force. . . . 

The  mission  of  the  BFVS  is  to  provide  mobile  protected  transport  of  an 
infantry  squad  to  critical  points  on  the  battlefield  and  to  perform  cavalry 
scout  missions.  The  BFVS  will  also  provide  overwatching  fires  to  support 
dismounted  infantry  and  suppress  or  defeat  enemy  tanks  and  other 
fighting  vehicles.  BFVS-A3  enhancements  include: 

•  Incorporation  of  Force  XXI  Battle  Command,  Brigade  and  Below 
Embedded  Battle  Command  to  share  command  and  control  and 
situation  awareness  with  all  components  of  the  combined  arms  team. 
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•  The  improved  Bradley  acquisition  system  and  commander  s 
independent  viewer, . . . ,  to  improve  target  acquisition  and  target 
engagement. 

•  A  position  navigation  system  with  a  Global  Positioning  System 
receiver  to  enhance  situational  awareness. 

The  following  comparison  of  the  Bradley  A2  and  the  Bradley  A3  is  provided  in  the 
LFT&E  Strategy  included  in  the  "Bradley  Fighting  Vehicle  System,  A3  Test  and 
Evaluation  Master  Plan  (TEMP)  [8]:" 

Although  much  of  the  ballistic  protection  is  the  same  as  on  the  Bradley 
A2,  some  changes  have  been  made  to  both  internal  and  external 
configurations:  electronic  systems  have  been  changed  and  upgraded, 
new  subsystems  have  been  added  to  the  vehicle  both  internally  and 
externally,  armor  covering  most  of  the  roof  and  some  of  the  upper 
portion  of  the  sides  has  been  improved,  a  pontoon  system  is  being 
developed  and  a  10th  crew  member  has  been  added. 

The  Bradley  A3  FU  SL  LFT,  an  element  of  the  vulnerability  assessment  plan  of  the 
system,  began  in  December  1998  at  ATC  in  Aberdeen  Proving  Ground,  MD,  and  is 
scheduled  for  completion  in  1999.  Eighteen  shots  are  planned  against  two  production 
qualification  test  (PQT)  vehicles,  upgraded  to  production  configuration  with 
3.0Z  software  end  product,  and  an  LRIP  vehicle. 


The  Bradley  A3  FU  SL  LFT  is  designed  to  investigate  the  effects  of  nonperforating 
and  perforating  threats  likely  to  be  encountered  in  combat.  The  critical  test  and 
evaluation  (T&E)  issues  are  defined  as  follows  in  the  LFT&E  Strategy  included  in  the 
"Bradley  Fighting  Vehicle  System,  A3  Test  and  Evaluation  Master  Plan  (TEMP)  [8]:" 

•  What  is  the  vulnerability  of  the  Bradley  A3  system  to  the  spectrum  of 
expected  threats  and  how  does  this  compare  to  previous  Bradley 
Fighting  Vehicles?  Subissues  [include:]  How  are  the  vulnerabilities  of 
the  Bradley  A3  changed  ...  by  the  additional  roof  armor?  What  impacts 
do  the  A3  modifications  have  on  Bradley  system  vulnerabilities?  ... 

What  is  the  impact  of  the  A3  modifications  on  the  Bradley  system's 
vulnerability  to  ballistic  shock? 

•  What  is  the  vulnerability  of  the  A3  crew  and  troops  to  the  spectrum  of 
expected  threats  and  how  does  this  compare  to  previous  Bradley 
Fighting  Vehicles?  Subissues  [include:]  How  is  the  vulnerability  of  the 
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Bradley  A3  crew  and  troops  changed  ...  by  the  additional  roof  armor? 

What  impacts  do  the  A3  modifications  have  on  Bradley  crew  and  troop 
vulnerability? ... 

•  How  effective  is  battle  damage  assessment  and  repair  (BDAR)  in 
restoring  the  A3  to  minimum  functional  combat  and  full  mission 
capability  following  a  hit  by  expected  threats  (direct  and  indirect)? 

Subissues  [include:]  What  changes  in  BDAR  procedures  are  required 
for  the  Bradley  A3  compared  to  the  Bradley  A2?  How  effective  are  the 
applicable  BDAR  manuals,  procedures,  tools,  and  supplies  for  the  A3? 

•  What  are  the  vulnerabilities  of  the  BFVS  A3  system/ crew/troops  to  new 
and/or  emerging  threats? 

•  Are  there  any  unexpected  BFVS  A3  vulnerabilities?  Subissues  [include:] 

What  is  the  operational  significance  of  any  unexpected  vulnerability? 

Can  these  vulnerabilities  be  reduced? 

2.2  Information  Sources.  The  following  agencies/organizations  provided 
information  for  this  research  study: 

•  Survivability /Lethality  Analysis  Directorate  (SLAD),  U.S.  Army  Research 
Laboratory  (ARL). 

•  Evaluation  Analysis  Center  (EAC),  U.S.  Army  Operational  Test  and  Evaluation 
Command  (OPTEC). 

•  Headquarters,  U.S.  Army  Test  and  Evaluation  Command  (HQ,  TECOM). 

•  U.S.  Army  Aberdeen  Test  Center  (ATC). 

•  U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA). 

•  PM  Offices  (PMOs): 

0  ATACMS  PMO,  U.S.  Army  Aviation  and  Missile  Command  (AMCOM), 
Redstone  Arsenal,  AL. 


*  In  a  recent  command  reorganization,  EAC,  OPTEC,  and  TECOM  became  elements  of  the  Army  Test  and 
Evaluation  Command  (ATEC). 
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0  Abrams  PMO,  U.S.  Army  Tank-automotive  and  Armaments  Command 
(TACOM),  Warren,  MI. 

0  Bradley  PMO,  TACOM,  Warren,  ML 
•  Institute  for  Defense  Analysis  (IDA). 

Discussions  with  the  personnel  from  SLAD,  EAC,  TECOM,  ATC,  and  AMSAA 
focused  on  general  background  information  of  the  respective  agency,  the  role  of  the 
agency  in  FU  SL  LFT&E  and  general  T&E  activities,  funding  and  budgeting  for  the 
activities  of  FU  SL  LFT&E  by  the  agency,  the  impacts  of  FU  SL  LFT&E  activities  of  the 
agency,  and  the  complexities  encountered  in  identifying,  measuring,  and  reporting 
those  impacts. 

Discussions  relative  to  the  application  of  the  V/L  Taxonomy  and  CATV  principles 
to  the  analysis  of  competing  V /L  assessment  strategies  were  an  important  part  of  the 
conversations  with  AMSAA,  EAC,  and  SLAD  personnel.  Limited  discussions  with  IDA 
personnel  focused  on  the  role  of  IDA  in  the  V/L  assessment  of  weapon  systems. 

Interviews  with  PMO  personnel  included  discussions  of  the  history  of  the  system 
associated  with  the  PMO,  congressional  funding  for  the  system,  contracts  associated 
with  various  cycles  of  the  acquisition  process  for  the  system,  the  role  of  the  PMO  in  the 
acquisition  process  and  the  FU  SL  LFT&E  of  the  system,  the  impacts  of  the  FU  SL 
LFT&E  of  the  system,  and  the  complexities  encountered  in  the  T&E  activities  of  the 
system. 

In  total,  more  than  55  scheduled  discussions  or  interviews  were  conducted  to  gather 
the  information  needed  for  this  research  project  with  approximately  60  people 
participating  in  these  conversations.  In  addition,  many  informal  discussions  occurred 
at  defense-related  conferences  and  workshops  attended  during  the  most  recent  2-  to 
3-year  period.* 

Conferences  and  workshops  included  LFT&E — 10  Years  and  Counting  Conference,  Lawrence 
Livermore  National  Laboratory,  Livermore,  CA,  January  1997;  Fourteenth  International  Symposium  on 
Military  Operational  Research,  Royal  Military  College  of  Science,  Shrivenham,  Wiltshire,  United 
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Many  documents  relative  to  the  three  systems,  LFT&E  and  its  impacts,  the  V/L 
Taxonomy,  the  CAIV  methodology,  T&E,  and  other  related  topics  were  reviewed. 

3.  Research  Results 

To  determine  the  impacts  of  FU  SL  LFT&E,  the  activities  that  constitute  the  FU  SL 
LFT&E  program  must  be  delineated.  In  reality,  conducting  an  Army  FU  SL  LFT&E 
program  results  in  activities  in  many  agencies,  including  the  Office  of  the  Director, 
Operational  Test  and  Evaluation  (DOT&E),  the  Office  of  the  Deputy  Director, 
Operational  Test  and  Evaluation,  Live-Fire  Testing  (DDOT&E  [LFT]),  the  Office  of  the 
Deputy  Under  Secretary  of  the  Army  for  Operations  Research  (DUSA  [OR]),  the  Office 
of  the  Deputy  Chief  of  Staff  for  Intelligence  (DCSINT),  the  U.S.  Army  Medical  Research 
and  Materiel  Command  (MRMC),  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  including  the  U.S.  Army  Ordnance  Center  and  School  (USAOC&S)  and  the 
U.S.  Army  Transportation  Center  and  School  (USATC&S),  the  U.S.  Congress  and  its 
subcommittees,  analysis  groups  utilized  by  the  aforementioned  offices  (e.g.,  IDA), 
TECOM,  SLAD,  ATC  and  other  test  centers,  AMSAA,  EAC,  and  the  PMOs  associated 
with  the  specific  weapon  system  tested.  For  the  purposes  of  this  research  project,  the 
FU  SL  LFT&E  program  is  defined  more  narrowly  and  the  activities  examined  are 
limited  to  those  described  in  the  following  section. 

3.1  Activities  of  FU  SL  LFT&E.  The  following  are  identified  as  the  activities 
completed  in  a  FU  SL  LFT&E.  Specific  divisions  and  agencies  with  lead  responsibilities 
for  the  activities  are  noted  as  appropriate. 


Kingdom,  September  1997;  Conference  on  the  Economics  of  Test  and  Evaluation,  TEREC,  Georgia 
Institute  of  Technology,  Atlanta,  GA,  October  1997;  Fourteenth  Annual  Test  &  Evaluation  Conference 
and  Exhibition,  San  Diego,  CA,  March  1998;  Fifteenth  International  Symposium  on  Military  Operational 
Research,  Royal  Military  College  of  Science,  Shrivenham,  Wiltshire,  United  Kingdom,  September  1998, 
Thirty-seventh  Annual  Army  Operations  Research  Symposium,  Fort  Lee,  VA,  October  1998;  Test  Plan 
Optimization  Workshop,  TEREC,  Georgia  Institute  of  Technology,  Atlanta,  GA  January  1999; 
Simulation  and  Modeling  for  Acquisition,  Requirements,  and  Training  Conference,  San  Antonio,  TX, 
January  1999;  10th  Annual  Army  Ground  Vehicle  Survivability  Symposium,  Monterey,  C  A,  March  1999; 
and  67th  Annual  Military  Operations  Research  Society  Symposium,  West  Point,  NY,  June  1999. 
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•  Formation  of  Live-Fire  Integrated  Product  Team  (LFIPT) 

EAC  leads  the  working  group  under  the  Test  and  Evaluation  Integrated  Product  Team; 
members  of  working  group  include  representatives  of  the  offices  ofDOT&E,  DUSA(OR), 
a  representative  of  the  military  intelligence  community  (e.g.,  National  Ground 
Intelligence  Center),  in  addition  to  personnel  from  EAC,  SLAD,  TECOM,  the  PMO,  the 
test  center,  USAOC&S/USATC&S,  and  TRADOC. 

•  Modeling  and  Simulation 

SLAD  performs  the  extension  or  expansion,  the  verification,  and  the  validation  of 
models  required  to  be  exercised  in  FU  SL  LFT&E  of  weapon  system;  exercises  the 
models  in  pre-shot  predictions;  and  provides  full-view  V/L  estimates  to  EAC  for 
use  in  the  FU  SL  LFT  evaluation. 

•  Development  of  LFT&E  Event  Design  Plan  (EDP)  that  Replaces  Prior 
Independent  Evaluation  Plan/Test  Design  Plan  (IEP /TDP) 

EAC  serves  as  lead  in  preparation  of  EDP;  receives  input  from  SLAD  and  other  members 
of  LFIPT. 

•  Writing  of  Detailed  LFT&E  Test  Plan  (DTP) 

Tester  prepares  DTP. 

0  Tester  of  ground  systems:  TECOM 
0  Tester  of  aviation  systems :  SLAD 
TECOM  reviews  DTP  of  LFT s  assigned  to  other  agencies  for  execution. 

•  Performance  of  FU  SL  LFT 

0  Setup  and  execution 

Designated  tester  sets  up  and  executes. 

0  Battlefield  damage  assessment  and  repair 
USAOC&S  conducts  the  BDAR  of  ground  systems. 

USATC&S  conducts  the  BDAR  of  aviation  systems. 

0  Damage  assessment  and  casualty  assessment 

SLAD  coordinates  damage  assessment  and  casualty  assessment. 
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•  Documentation 

0  Damage  assessment  shot  records 

SLAD  prepares  damage  assessment  shot  records. 

0  Detailed  Damage  Assessment  Report 

SLAD  prepares  Detailed  Damage  Assessment  Report. 

0  Detailed  Test  Report  (DTR) 

Tester  serves  as  lead  in  preparation  of  DTR,  receiving  input  from 
Damage  Assessment  Team ;  TECOM  reviews  DTR  for  LFTs  assigned  to 
other  agencies  for  execution. 


Independent  Evaluation  and  Preparation  of  System  Evaluation  Report  (SER), 
Including  the  Analysis  of  LFT&E  Activities 
EAC  prepares  the  SER. 


3.2  Costs  of  FU  SL  LFT&E.  A  weapon  system  included  in  the  U.S.  congressional 
LFT&E  mandate  must  conduct  a  FU  SL  LFT&E  as  part  of  the  assessment  of  the 
survivability/lethality  program  of  the  system,  unless  the  waiver  requested  by  the 
Service  acquiring  the  system  is  granted  by  the  Secretary  of  Defense.  The  cost  of  the  FU 
SL  LFT&E  is  an  important  consideration  in  determining  the  appropriateness  of  a 
waiver,  in  budgeting  dollars  for  the  execution  of  a  FU  SL  LFT&E,  and  in  comparing 
various  V/L  assessment  strategies.  Perhaps  the  most  difficult  issue  faced  in 
determining  the  cost  of  FU  SL  LFT&E  is  defining  the  term  cost.  The  analyst  must 
determine  what  components  of  the  FU  SL  LFT&E  activities  (listed  in  the  previous 
paragraph)  should  be  costed  and  how  the  costs  of  those  components  should  be 
measured.  In  this  process,  the  analyst  must  address  the  following  questions: 


•  How  are  the  costs  of  the  tested  systems  (i.e.,  test  articles)  computed,  if  systems 
are  damaged  but  salvageable?  How  are  the  costs  computed,  if  the  test  articles 
are  completely  destroyed? 

•  How  are  the  costs  of  spare  parts  used  in  repairs  computed? 
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•  How  are  the  costs  of  targets  used  in  lethality  testing  or  munitions  used  in 
vulnerability  testing  computed? 

•  How  are  the  labor  costs  of  individuals  participating  in  the  planning;  modeling 
and  simulation  (M&S);  testing;  or  evaluation  process  of  FU  SL  LFT&E  computed? 

•  How  are  the  costs  of  the  test  ranges  and  instrumentation  used  in  FU  SL  LFTs 
computed  or  allocated  across  the  many  systems  tested  at  the  test  center  or  across 
the  multiple  time  periods  over  which  the  test  center  is  in  operation? 

To  make  a  valid  comparison  of  assessment  strategies  (e.g.,  those  that  include  FU  SL 
LFT&E  and  those  that  do  not),  decision-makers  must  be  provided  with  cost  data  that 
are  identified  and  measured  according  to  a  consistent  methodology  employed  across 
assessment  plans.  Clear  descriptions  of  FU  SL  LFT&E  activities  and  the  separate 
components  of  each  activity  and  their  related  costs  must  be  identified.  In  situations  in 
which  there  are  alternative  ways  of  measuring  the  costs  of  an  activity,  the  measurement 
method  employed  should  be  identified  and  described.  Allocated  costs  should  be  so 
identified,  and  the  bases  of  allocation  or  the  rates  used  in  allocation  should  be  fully 
explained.  In  addition,  information  that  allows  the  decision-maker  to  compute  the  costs 
in  an  alternative  manner  should  be  disclosed  by  the  cost  analysts. 

For  purposes  of  this  research  study,  cost  sheets  of  the  completed  FU  SL  LFT  of  both 
the  ATACMS  Block  LA  and  the  Abrams  Ml  A2  were  provided  by  the  ATACMS  and 
Abrams  PMOs  and  TECOM.  The  cost  sheet  of  the  1993  Abrams  Ml  A2  included  costs 
aggregated  in  five  very  broad  categories  (i.e..  Test  Center,  Repair  Parts,  ARL/ 

TECOM/ AMS AA,  Ammunition  and  Nondestructive  Tests,  and  Contractor  Test 
Support).  The  time  period  between  the  completion  of  this  test  and  the  changes  in 
personnel  across  agencies  made  the  costs  related  to  the  Abrams  M1A2  difficult  for 
interviewees  to  re-construct.  The  cost  sheet  of  the  more  recent  1996  FU  SL  LFT&E  of  the 
ATACMS  Block  IA  included  a  more  detailed  breakdown  of  the  costs  related  to  work 
performed  by  SLAD  and  the  two  test  centers  (i.e.,  ATC  and  WSMR)  in  addition  to  some 
target-related  costs.  Costs  for  the  forthcoming  Bradley  A3  FU  SL  LFT&E  were  available 
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only  in  total  budgeted  amounts  for  the  tests.  All  costs  referred  to  in  the  following 
sections  were  taken  from  the  sheets  provided  for  this  study. 

Personnel  from  all  three  PMOs  reported  that  records  are  kept  of  the  FU  SL  LFT&E 
costs  incurred  by  government  agencies  and  by  contractor(s)  and  subsequently 
reimbursed  by  their  offices.  ATC  reports  that  it  provides  the  PMO  with  both  interim 
reports  of  the  costs  incurred  by  the  test  center  (i.e.,  during  the  execution  of  the  FU  SL 
LFT)  and  a  final  accounting  of  test-related  costs  at  the  conclusion  of  the  FU  SL  LFT.  The 
PMO,  however,  is  not  required  to  produce  a  formal  report  detailing  the  total  costs  of  all 
FU  SL  LFT&E-related  costs  incurred  by  government  agencies  and  contractors  and 
reimbursed  by  the  PMO. 

This  research  project  did  not  have  the  objective  of  critically  reviewing  the  costs 
incurred  or  reported  by  the  systems  studied  but  of  understanding  how  the  costs  of  FU 
SL  LFT&E  are  identified,  measured,  and  reported  and  what  complexities  are 
encountered  in  accomplishing  these  tasks.  An  examination  of  the  information  included 
in  the  available  cost  sheets  and  discussions  with  individuals  associated  with  the 
activities  of  FU  SL  LFT&E  provided  much  insight  into  these  issues. 

In  this  report,  the  costs  of  FU  SL  LFT&E  are  placed  in  one  of  two  categories: 

(1)  costs  of  direct  materials  used  and  consumed  in  the  FU  SL  LFT&E  or  (2)  costs  of 
activities,  including  the  costs  of  indirect  materials,  labor,  and  facilities  used  in  the 
planning,  execution,  and  evaluation  of  the  FU  SL  LFT  and  its  results. 

3.3  Costs  of  FU  SL  LFT&E:  Costs  of  Direct  Materials.  The  costs  of  direct 
materials  used  or  consumed  in  the  FU  SL  LFT&E  are  defined  as  all  material  costs  that 
can  be  feasibly  or  economically  traced  to  the  cost  objective  (i.e.,  FU  SL  LFT&E).  The 
direct  material  costs  of  FU  SL  LFT&E  include  the  costs  of  the  following  test  assets: 

(1)  the  test  articles  (i.e.,  weapon  systems  tested),  (2)  the  spare  parts  provided  for  the  test 
articles,  (3)  the  munitions  fired  (i.e.,  applicable  to  vulnerability  tests),  and  (4)  the  targets 
fired  upon  (i.e.,  applicable  to  lethality  tests).  It  is  assumed  in  this  report  that  the  total 
cost  of  all  test  assets  discussed  will  include  the  transportation  and  preparation  costs 
incurred  to  ready  the  asset  for  FU  SL  LFT  at  the  designated  test  center. 
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3.3.1  Cost  of  Test  Articles.  The  following  costs  are  proposed  as  alternative 
reporting  values  for  test  articles  used  in  FU  SL  LFT&E: 

•  Cost  of  replacing  the  test  article  with  an  article  identical  to  the  original  article 
(i.e.,  replacement  cost). 

•  Cost  of  returning  the  test  article  following  FU  SL  LFT  to  its  condition  prior  to 
FU  SL  LFT  (i.e.,  restorative  cost). 

•  Cost  of  acquiring  the  original  test  article  (i.e.,  historical  cost). 

The  costs  of  test  articles  have  been  accounted  for  in  various  ways  in  completed  FU 
SL  LFT&E  programs.  For  example,  the  total  program  costs  for  the  FU  SL  LFT&E 
provided  by  the  Abrams  PMO  do  not  include  the  costs  of  the  vehicles  used  in  testing. 
Both  test  articles  were  deemed  salvageable  at  the  conclusion  of  the  1993  LFT  program 
with  an  expenditure  of  $1.5  million  estimated  to  return  the  two  Abrams  Ml  A2  LRIP 
vehicles  to  combat  condition.  The  PMO  requests  for  dollars  to  return  the  articles  to 
combat  condition  were  not  funded,  and  the  two  tanks  are  currently  in  use  as  training 
vehicles  (e.g.,  track  tests)  at  APG.  Replacement  cost  or  the  current  cost  to  convert  an 
Abrams  Ml  to  an  M1A2  (cost  includes  some  engineering  support  for  potential 
production  problems)  is  estimated  by  the  Abrams  PMO  at  $4-5  million  per  vehicle.' 
Historical  costs  or  the  actual  costs  incurred  to  construct  the  two  LRIP  Abrams  M1A2 
vehicles  used  in  the  FU  SL  LFT&E  were  not  available. 

Neither  the  cost  of  the  missiles  used  in  the  ATACMS  Block  LA  tests  (i.e.,  launcher- 
to-target  tests)  nor  the  relatively  small  cost  of  the  M74  bomblets  (i.e.,  approximately  100 
bomblets  used  at  $30/bomblet)  in  the  arena  tests  was  included  on  the  LFT&E  cost  sheet 
produced  by  the  ATACMS  PMO.  PMO  personnel  stated  that  the  missile  cost  was 
treated  by  its  office  as  part  of  the  costs  of  the  developmental  and  operational  tests  of  the 
ATACMS  Block  LA.  The  PMO  quoted  a  figure  of  $1.01  million  as  the  cost  of  producing 


*  It  should  be  noted  that  this  is  the  replacement  cost  estimate  in  1998;  replacement  cost  in  1993  is 
unavailable. 
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an  ATACMS  Block  IA  missile  (i.e.,  replacement  cost).  This  cost  includes  the  costs  of 
contractor,  support  of  PM  office,  and  production  testing. 


The  projected  cost  figures  for  the  LFT&E  program  of  the  Bradley  A3  include 
$2.7  million  (of  the  total  $6  million  FU  SL  LFT&E  budget  of  the  Bradley  PMO)  budgeted 
for  the  replacement  of  two  PQT  vehicles. 


From  the  aforementioned  three  examples,  it  is  evident  that  the  total  costs  of  FU  SL 
LFT&E  may  be  significantly  affected  by  the  choice  of  the  alternative  reporting  value 
chosen  for  the  cost  of  the  test  article(s).  It  is  proposed  in  this  report  that  the  restorative 
cost  (i.e.,  the  cost  to  return  the  test  article  to  its  original  condition)  is  the  most 
appropriate  value  to  report  as  the  test  article  cost  if  (1)  the  test  article  is  in  a  salvageable 
condition  following  testing,  (2)  the  restorative  cost  is  able  to  be  estimated,  and  (3)  the 
restorative  cost  is  the  smallest  of  the  three  proposed  alternative  values.  The  restorative 
cost  is  a  cost  of  testing,  regardless  of  whether  or  not  the  expenditure  is  made.  If  the 
restorative  cost  is  higher  than  one  or  both  of  the  remaining  alternative  values  (i.e., 
replacement  cost  and  historical  cost)  or  the  test  article  cannot  be  restored,  a  choice  must 
be  made  as  to  which  of  the  two  alternatives  will  be  reported  as  the  test  article  cost.  Each 
value  has  its  own  merits,  but  it  is  suggested  that  the  lower  of  the  two  values  be  reported 
as  the  cost  of  the  test  article.*  Regardless  of  which  value  is  reported,  all  available 
alternative  values  for  costing  the  test  article  should  be  disclosed  in  the  cost  report  and 
made  part  of  the  database  available  to  those  budgeting  for  FU  SL  LFT&E  of  future 
systems.  The  computations  of  all  reported  values,  including  the  amounts  attached  to 

*  An  LRIP  vehicle  was  added  to  the  test  artides  planned  to  be  used  in  the  FU  SL  L.FT.  The  PMO  stated 
that  all  Bradley  A3  vehides  were  expected  to  be  remanufactured  from  used  vehicles,  and  there  were  no 
current  plans  to  build  any  new  BFVS  vehicles  for  the  U.S.  Army. 

+  In  general,  an  asset  is  reported  at  its  historical  cost  under  generally  accepted  accounting  principles 
because  the  historical  cost  is  assumed  to  be  objectively  determined  in  an  arm's  length  transaction 
between  two  independent  parties  and  therefore  refleds  the  fair  value  of  the  asset  at  the  transaction  da  e. 
In  the  case  of  FU  SL  LFT&E,  however,  the  historical  cost  of  the  prototype  test  asset  used  in  FU  SL  LM 
could  be  relatively  high  in  comparison  to  the  replacement  cost  of  the  asset.  It  is  hard  to  defend  the 
position  that  if  the  test  asset  is  lost  in  the  FU  SL  LFT,  the  entire  historical  cost  of  the  prototype  should  be 
reported  as  a  cost  of  FU  SL  LFT&E  instead  of  reporting  the  replacement  cost  of  the  asset.  In  reality,  a 
system  would  not  be  expected  to  be  restored  IF  its  replacement  cost  were  lower  than  its  restorative  cost 
(i  e.,  given  enough  time  for  replacement).  Therefore,  the  suggested  rule  reduces  to  report  the  lowest  of 
the  three  values  available:  restorative  cost,  replacement  cost,  or  historical  cost. 
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these  costs  (e.g.,  engineering  support  costs),  should  be  explained  in  full  to  facilitate  the 
processing  of  this  information  in  comparative  analyses  of  assessment  plans  and  in 
future  budgeting  activities. 

The  costs  of  assets  other  than  the  test  articles  include  the  costs  of  the  spare  parts 
provided  for  the  repairs  of  the  test  articles  made  during  the  FU  SL  LFT&E,  the 
munitions  fired  (i.e.,  vulnerability  tests),  and  the  targets  fired  upon  (i.e.,  lethality  tests). 
As  in  the  cost  of  test  articles,  there  are  often  alternative  values  (e.g.,  historical  cost, 
restorative  cost,  and  replacement  cost)  available  for  assets  included  in  this  category. 

3.3.2  Cost  of  Spare  Parts.  The  accounting  for  spare  parts  for  test  articles  in  the  FU 
SL  LFTs  of  two  of  the  systems  studied  raises  the  question,  "Are  spare  parts  provided  to 
the  test  center  as  part  of  a  larger  contract,  and  are  those  costs  able  to  be  separated  from 
the  major  costs  of  the  contract  and  reported  as  a  line  item  in  a  FU  SL  LFT&E  cost 
report?"*  In  the  Abrams  M1A2  FU  SL  LFT&E,  the  spare  parts  were  provided  at  a  fixed 
cost  as  a  test  system  support  package  attached  to  the  LRIP  Contract  for  62  tanks  with 
General  Dynamics  Land  Systems,  and  ATC  accounted  for  the  spare  parts  as  used.  The 
Abrams  FU  SL  LFT&E  cost  sheet  reported  a  cost  of  $1.6  million  for  repair  parts,  but  the 
Abrams  PMO  stated  this  estimate  could  be  as  high  as  $2  million.  The  attachment  of  the 
spare  parts  cost  package  to  the  LRIP  contract  made  this  cost  more  difficult  to  estimate. 

The  Bradley  PMO  reported  that  it  plans  to  acquire  spare  parts  needed  for  the  FU  SL 
LFT&E  of  the  Bradley  A3  from  Army  depots  (i.e.,  Bradley  common  parts)  and  the  LRIP 
contractor.  United  Defense  Limited  Partners  (i.e.,  Bradley  A3  unique  parts).  Pre-shot 
predictions  have  allowed  the  PMO  to  project  parts  that  will  be  needed  in  the  FU  SL 
LFT.  The  LRIP  contractor  will  provide  the  parts  as  needed  from  vehicles  completed  in 
its  contract,  and  a  contract  modification  will  be  added  following  the  completion  of  the 
FU  SL  LFT  to  account  for  parts  required  during  the  testing  phase. 


Costs  for  spare  parts  of  test  assets  are  not  applicable  to  the  lethality  tests  of  the  ATACMS  Block  IA  (with 
the  exception  of  launcher  spare  parts). 
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The  cost  incurred  at  the  time  of  purchase  (i.e.,  historical  cost)  is  generally  reported 
as  the  cost  of  spare  parts.  In  addition,  the  reporting  of  this  cost  should  include  a 
description  of  the  contract  agreement  (e.g.,  pay  contractor  a  fixed  amount  for  all  parts 
needed,  pay  contractor  an  amount  that  varies  with  the  type  and  number  of  spare  parts 
needed,  etc.).  Disclosure  of  the  spare  parts  used  vs.  spare  parts  purchased  (i.e.,  in  fixed- 
dollar  contract)  and  related  costs  may  prove  useful  in  planning  of  subsequent  FU  SL 
LFT  events. 

3.3.3  Cost  of  Munitions  and  Targets.  The  costs  of  some  assets  related  to  FU  SL 
LFT&E  are  particularly  difficult  to  determine.  The  costs  of  munitions  used  in 
vulnerability  tests  or  targets  used  in  lethality  tests  may  not  be  readily  available  or  easily 
estimated.  Munitions  and  targets  may  have  been  obtained  without  exchanging  dollars, 
if  acquired  in  prior  combat  or  in  nonmonetary  trades  with  other  agencies.  ATC  reports 
that  munitions  are  often  free  or  obtained  at  a  relatively  insignificant  cost  from  a  foreign 
source. 

No  cost  information  relative  to  the  munitions  used  in  the  completed  Abrams  Ml  A2 
FU  SL  LFT&E  (i.e.,  vulnerability  test)  is  included  on  the  cost  sheet  provided.  Neither 
the  costs  of  the  targets  used  in  the  launcher-to-target  and  arena  tests  of  the  ATACMS 
Block  IA  (i.e.,  lethality  tests)  nor  the  costs  of  the  repairs  to  the  targets  used  in  those  tests 
were  totally  accounted  for  on  the  FU  SL  LFT&E  cost  sheet  provided.  Additional  targets 
in  the  launcher-to-target  tests  conducted  at  WSMR  were  required  and  funded  by  OSD 
and  were  not  included  in  the  costs  reported  on  the  cost  sheet  as  reimbursable  by  the 
PMO.  The  cost  sheet  does  include  a  portion  of  the  cost  of  transportation  for  targets 
used  in  the  arena  tests. 


Target  repair  costs  were  not  included  on  the  ATACMS  Block  IA  cost  sheet.  The  cost 
sheet  does  not  include  the  sum  requested  by  the  Project  Manager,  Instrumentation, 


*  Of  the  3  targets  used  in  the  arena  tests  of  the  ATACMS  Block  IA,  only  1  was  a  target  included  in  the 
Operational  Requirements  Document  (ORD)  list  of  11  targets;  the  other  2  targets  had  characteristics 
similar  to  the  other  10  ORD  target  types.  Targets  used  in  the  launcher-to-target  tests  included  some 
target  surrogates,  some  foreign  targets  with  elements  representative  of  ORD  targets,  and  some  targets 
that  were  less  vulnerable  than  ORD  targets. 
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Targets  and  Threat  Simulators  (point  of  contact  for  obtaining,  controlling,  operating, 
and  sustaining  threat  equipment  or  surrogates  for  use  in  LFT)  for  restoring  the  targets 
used  in  the  ATACMS  Block  LA  FU  SL  LFT.  The  PMO  reported  the  intent  to  pursue 
some  "trading"  of  other  target  assets  with  the  Target  Management  Office  to  compensate 
the  Office  for  damage  sustained  to  borrowed  targets.  It  may  be  difficult  to  obtain  an 
historical  cost  or  to  estimate  a  replacement  cost  for  the  munitions  used  in  lethality  tests 
or  the  targets  used  in  vulnerability  tests.  A  complete  accounting  of  the  costs  of  test 
assets,  however,  should  include,  if  available,  an  estimated  cost  of  the  munitions /targets 
with  the  relevant  range  of  values  for  that  estimated  cost  or  a  disclosure  of  the  inability 
to  estimate  the  cost  for  identified  assets. 

3.4  Costs  of  FU  SL  LFT&E:  Costs  of  Activities.  In  this  report,  the  costs  of 
activities  of  FU  SL  LFT&E  are  defined  as  the  costs  incurred  for  indirect  materials 
consumed,  labor  performed,  and  facilities  used  in  the  planning,  execution,  and 
evaluation  of  a  FU  SL  LFT  and  its  results.  To  compute  the  cost  of  an  activity,  all 
components  of  the  activity  that  consume  resources  and  the  costs  associated  with  those 
components  must  be  identified.  The  agencies  associated  with  the  activities  of  FU  SL 
LFT&E  described  in  the  beginning  of  this  section  account  for  their  costs  in  different 
ways  as  shown  in  the  following  sections. 

Labor  costs  are  a  large  component  of  the  costs  of  all  FU  SL  LFT&E  activities  across 
all  agencies.  In  accounting  for  the  labor  costs  of  employees  engaged  in  activities  in  an 
agency,  the  following  questions  must  be  addressed: 

•  Do  labor  costs  include  overhead  administrative  costs  of  the  employee’s  agency? 

•  Are  labor  costs  based  on  the  actual  hours  worked  or  the  hours  estimated  to  be 
needed  for  the  required  service?  In  other  words,  is  the  cost  of  labor  reported  in 
fact  the  budgeted  cost  of  labor,  similar  to  the  estimated  or  budgeted  cost  of  labor  in 
a  contract  that  is  fulfilled  with  the  delivery  of  the  product? 

3.4.1  Cost  of  SLAD  (ARL)  Activities.  There  are  two  major  SLAD  activities: 

(1)  M&S  and  (2)  damage  and  casualty  assessment,  in  addition  to  LFIPT  participation. 
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A  closer  look  at  the  SLAD  M&S  activity  reveals  four  components:  (1)  extension 
and/or  expansion  of  models  to  include  descriptions  and  characteristics  of  the 
test  article,  munitions,  target(s)  of  interest,  and  damage  mechanisms  not  previously 
modeled,  in  addition  to  improvements  in  damage  mechanisms  already  part  of  the 
model  (e.g.,  behind-armor  debris);  (2)  verification  and  validation  (V&V)  of  existing 
model(s);  (3)  exercise  of  model  in  pre-shot  predictions  and  reruns  of  pre-shot 
predictions  (e.g.,  reruns  needed  if  actual  geometry  of  shots  is  different  than  planned 
shots);  and  (4)  provision  of  full-view  V /L  estimates  to  E AC  for  use  in  the  FU  SL  LFT 
evaluation.  Modeling  activities  related  to  pre-shot  predictions  provide  important  data 
for  decisions  related  to  test  planning  and  execution  (e.g.,  selection  of  shot  lines,  stock 
levels  of  spare  parts,  etc.)  and  a  certain  measure  of  quality  control  for  the  modeling 
function.  On  the  other  hand,  modeling  activities  related  to  the  production  of  full-view 
V/L  estimates  generate  data  important  to  the  evaluation  of  the  FU  SL  LFT  and  the 
overall  assessment  of  the  V/L  of  the  system. 

As  stated  previously,  the  cost  sheet  for  the  FU  SL  LFT&E  of  the  ATACMS  Block  IA 
system  includes  a  more  detailed  accounting  of  costs  than  is  available  for  the  other  two 
systems  studied  and  is  therefore  used  as  an  example  of  the  costing  of  SLAD  M&S 
activities.  The  FU  SL  LFT  of  the  ATACMS  Block  IA  incurred  costs  for  the  development 
of  the  inputs  to  the  model  including  target  descriptions,  criticality  analyses,  and  the 
probability  of  component  kill  assessments  for  the  three  targets  of  the  arena  tests.  Pre¬ 
shot  predictions  were  also  accomplished  for  the  23  planned  shots  against  the  3  arena 
targets.  Following  the  post-shot  analysis,  the  vulnerability  analysis  was  updated  with  a 
rerun  of  the  SAFE  model  including  the  target  description  updates  (9  targets  included  in 
Operational  Requirements  Document  [ORD]).  As  part  of  the  follow-on  effort, 
algorithms  were  developed  to  characterize  damage  mechanisms  and  reactions  not 
previously  modeled,  such  as  pyrophonic  and  fragment  effects  given  fuel  and  propellant 
interaction. 

Verification,  validation,  and  accreditation  (W&A)  of  models  exercised  as  part  of 
the  FU  SL  LFT&E  must  be  achieved  prior  to  relying  on  modeling  efforts  in  FU  SL 
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LFT&E.*  SAFE  and  Stochastic  Quantitative  Analysis  of  System  Hierarchies  (SQuASH) 
models,  as  exercised  in  the  FU  SL  LFT&E  of  ATACMS  Block  IA,  have  recently 
undergone  model  calibration  exercises  in  which  the  models  are  compared  to  existing 
test  results.  The  costs  of  these  exercises  for  the  ATACMS  Block  IA  FU  SL  LFT&E  were 
absorbed  by  SLAD  mission  funds.  The  FU  SL  LFT&E  of  the  Abrams  M1A2  relied  on  the 
results  of  model  FORTRAN  SQuASH,  but  W&A,  as  it  is  now  known,  was  not 
performed  for  the  Abrams  system  in  1993.  The  SQuASH  and  SAFE  sections  of  the 
Modular  UNIX-based  Vulnerability  Estimation  Suite  (MUVES)  have  been  consolidated 
into  a  new  combination  model,  MUVES-S2.  The  V&V  of  MUVES-SQuASH  for  Bradley 
A2  has  been  completed  by  SLAD  in  preparation  for  the  Bradley  A3  LFT,  but  V&V  for 
SAFE  relative  to  Bradley  A3  is  ongoing.  The  costs  of  the  effort  for  V&V  required  for 
Bradley  A3  have  again  been  absorbed  by  SLAD  mission  funds.  SLAD  personnel  expect 
that  in  the  future  PMOs  will  be  asked  to  contribute  to  the  V&V  effort,  including  the 
reimbursement  of  material  and  labor  costs  incurred  in  the  experiments  and  analyses 
necessary  to  support  the  changes  in  codes  and  algorithms  associated  with  the  models 
required  for  a  system's  FU  SL  LFT&E. 

The  damage  assessment  team,  generally  chaired  by  SLAD  personnel,  is  responsible 
for  the  damage  assessment  at  the  conclusion  of  each  FU  SL  LFT  shot.  Members  of  the 
team  generally  include  personnel  from  SLAD  and  the  test  center  and  may  include 
personnel  from  the  proponent  TRADOC  school  and  EAC  among  others.  The  damage 
assessment  shot  records,  in  which  the  damage  assessments  for  all  FU  SL  LFT  shots  are 
detailed,  as  well  as  ancillary-related  data,  are  prepared  by  SLAD  and  test  center 
personnel  for  inclusion  in  the  DTR  produced  by  the  test  center.  The  Damage 
Assessment  Report,  which  includes  the  comparisons  of  the  results  of  the  FU  SL  LFT 
shots  with  pre-shot  predictions,  is  also  prepared  by  SLAD  personnel. 

The  personnel  casualty  assessment  effort  is  also  led  by  SLAD  personnel  with 
support  from  the  test  center  and  others  as  needed  (e.g.,  personnel  from  MRMC, 


'  Accreditation  of  models  is  not  performed  by  Survivability/Lethality  Analysis  Directorate  (SLAD) 
personnel. 
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including  Walter  Reed  Army  Institute  of  Research  and  Aeromedical  Research 
Laboratory).* 

Examples  of  the  costs  incurred  by  SLAD  in  damage  and  personnel  casualty 
assessments  can  be  found  on  the  cost  sheet  for  the  FU  SL  LFT&E  of  the  ATACMS 
Block  IA  system.  Included  are  the  costs  for  damage  assessment  at  ATC  and  WSMR, 
damage  assessment  support  and  model  verification,  and  personnel  casualty  assessment 
at  ATC  and  WSMR.  SLAD  employee-years  budgeted  by  branch  chiefs  include  the 
participation  of  branch  employees  in  system-related  working  group  meetings.  SLAD 
also  budgets  for  its  labor  costs  incurred  in  the  support  of  the  tester's  plans  and  reports. 
For  example,  the  ATACMS  FU  SL  LFT&E  cost  sheet  included  dollar  figures  for 
IEP/TDP  support,  preparation  of  DTP,  DTR,  General  Support,  and  Final  Test  Report. 

A  SLAD  branch  chief  budgets  labor  hours  for  FU  SL  LFT&E  activities  by  identifying 
the  employees  and  the  number  of  employee-years  needed  by  each  identified  employee 
in  the  branch  to  complete  the  tasks  required  for  the  LFT  effort.  Labor  cost  per 
employee-year  is  multiplied  by  the  number  of  employee-years  estimated  for  the  LFT- 
related  task,  and  this  product  becomes  the  total  estimated  costs  for  that  employee  on  the 
identified  task. 

A  labor  cost  per  employee-year  for  each  branch  employee  and  an  average  branch 
labor  cost  per  employee-year  are  computed  by  the  SLAD  division  and  directorate 
offices  and  provided  to  each  branch  chief  for  the  purposes  of  budgeting  and  costing 
services  rendered  by  branch  employees.  The  employee-year  cost  computation  includes 
the  salary  and  fringe  benefits  of  the  employee  plus  an  amount  (based  on  rate  per 
employee-hours  worked)  to  cover  overhead  associated  with  division  and  directorate 
administrative  offices.+  Projections  for  SLAD  costs  related  to  LFT  are  made  often  with  a 
5-year  horizon,  but  estimates  are  generally  revised  on  an  annual  basis. 

*  Most  test  centers  provide  some  form  of  data  collector  support  (e.g.,  photography  and  assistance  with 
post-shot  component  and  subsystem  checks).  Generally,  ATC  is  enlisted  to  perform  data  reduction  and 
analyses  for  damage  and  personnel  casualty  assessment. 

+  The  SLAD  overhead  hourly  rate  has  three  components,  including  the  direct  and  indirect  overhead  rates 
of  the  division  office  and  the  general  and  administrative  rate  of  the  directorate  office. 
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In  addition  to  the  costs  incurred  in  LFT&E  activities  reviewed  in  preceding 
paragraphs,  SLAD  employees  also  incur  travel  costs  related  to  LFT&E  activities.  By 
law,  the  PMO  of  the  tested  system  is  responsible  for  the  reimbursement  of  all  costs 
related  to  the  FU  SL  LFT&E  of  a  system,  including  travel-related  costs,  with  the 
exception  of  costs  incurred  by  agencies  with  mission-related  responsibilities  in  the 
LFT&E  program.  As  previously  mentioned,  some  of  the  FU  SL  LFT  costs  related  to 
M&S  have  been  financed  with  mission  funds  of  SLAD  (e.g.,  V&V  of  models  for  Bradley 
A3  FU  SL  LFT&E).  SLAD  mission  funds  are  delegated  to  systems  at  the  beginning  of 
the  year,  based  on  the  priorities  of  the  systems  for  which  the  SLAD  Mission  Area 
Manager  (MAM)  has  responsibility  in  the  period. 

SLAD  personnel  report  that  the  variance  or  difference  between  the  number  of 
employee-hours  budgeted  (basis  for  request  for  PMO  reimbursement)  and  the  number 
of  employee-hours  actually  used  to  complete  a  FU  SL  LFT&E  task  is  not  tracked  in 
detail.  If,  however,  there  are  significant  changes  made  in  the  tasks  required  of  SLAD 
employees,  the  budgeted  reimbursements  are  adjusted  to  reflect  the  difference  in  hours 
needed  for  completion  of  assigned  tasks.  Labor  costs  associated  with  the  additional 
hours  required  are  funded  frequently  with  a  combination  of  PMO  and  mission  dollars. 

Costs  incurred  for  materials  (e.g.,  experimental  materials  in  modeling)  required  in 
SLAD  tasks  are  included  in  dollar  amounts  requested  by  SLAD  for  PMO 
reimbursement.  The  significant  portion  of  the  SLAD  costs,  however,  is  for  employee 
services  (including  the  overhead  administrative  costs  related  to  SLAD  division  and 
directorate  offices). 

3.4.2  Cost  of  Test  Center  Activities.  There  are  three  major  LFT  activities  performed 
by  the  test  center:  (1)  planning  for  the  FU  SL  LFT,  including  preparation  of  the  DTP, 

(2)  execution  of  the  FU  SL  LFT,  including  instrumentation  and  target  repair  and 
maintenance,  and  (3)  preparation  of  the  DTR.  In  addition,  test  center  personnel  serve  as 
members  of  the  LFIPT  led  by  EAC  and  the  damage  assessment  team  led  by  SLAD.  The 
test  center  assembles  a  dedicated  test  team,  headed  by  the  test  director  and  specifically 
trained  in  the  operation,  maintenance,  and  repair  of  the  test  articles.  The  test  team 
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indudes  data  collectors  who  provide  the  required  test  data  to  the  test  director  for 
indusion  in  the  test  report. 

The  test  center  typically  is  involved  in  planning  activities  for  several  years  prior  to 
the  actual  execution  of  the  FU  SL  LFT.  Its  planning  activities  include  developing  or 
procuring  unique  instrumentation  as  needed  for  test  execution,  investigating  and 
developing  techniques  to  fire  unique  and/or  foreign  ammunition  in  accordance  with 
LFT  objectives,  coordinating  training  necessary  for  test  execution  events,  and  handling 
logistics  of  obtaining  and  maintaining  test  assets,  spare  parts,  etc.  As  part  of  its 
partidpation  in  the  planning  for  the  LFT  program,  the  test  center  provides  cost  and  time 
estimates  for  each  of  the  test  center  activities  that  make  up  the  FU  SL  LFT&E. 

As  an  example  of  the  costs  of  a  (TECOM)  test  center,  ATC  costs  are  discussed  in  the 
following  paragraphs.  The  FU  SL  LFTs  of  the  Abrams  M1A2  and  the  ATACMS  Block 
LA  (arena  tests)  were  conducted  at  ATC  at  APG.  The  Bradley  A3  FU  SL  LFT,  begun  at 
ATC  during  the  month  of  December  1998,  is  scheduled  to  be  completed  in  July  1999.  In 
addition  to  test  ranges,  ATC  provides  technical  shops,  performance  measurement 
laboratories,  and  instrumentation  and  analysis  laboratories.  Employing 
instrumentation  designed  to  evaluate  crew  survivability,  ATC  personnel  measure  crew 
incapacitation  parameters  as  part  of  the  tests  of  vehide  vulnerability. 

The  assigned  ATC  test  director  has  the  responsibility  to  develop  a  test  plan  that 
meets  the  technical  objectives  of  the  customer  at  the  least  possible  costs.  In  preparing  its 
cost  estimates  for  the  PMO  customer,  ATC  reports  that  it  conducts  a  detailed  analysis  of 
labor  and  material  costs,  relying  on  historical  cost  data,  expert  opinion,  and  relevant 
cost  estimate  methodologies. 

ATC  is  reimbursed  by  the  PMO  for  costs  incurred  in  FU  SL  LFT  on  the  bases  of  the 
activities  required  by  the  LFT  program  (e.g.,  setup,  repair,  analysis,  completing  test 
report)  and  the  costs  of  materials  and  services  required  to  complete  those  activities.  In 
computing  the  expected  costs  for  a  specific  activity  to  be  performed  as  part  of  the  FU  SL 
LFT  program,  the  ATC  project  manager  (1)  identifies  all  tasks  and  subtasks  required  to 
complete  the  activity,  (2)  determines  the  skill  (e.g.,  data  quality,  international  imaging. 
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welding)  and  skill  level  required  to  complete  the  identified  tasks,  and  (3)  projects  the 
number  of  employee-hours  needed  to  complete  all  identified  tasks.  Using  the  project 
management  software,  the  cost  estimate  for  the  FU  SL  LFT  program  is  established  by 
summing  across  all  required  tasks  (1)  the  respective  products  of  estimated  hours 
projected  for  each  skill  and  average  wage  rate  established  for  the  identified  skill  and 
(2)  the  other  nonlabor  costs  incurred  in  the  task. 

ATC  charges  the  PMO  customer  for  labor  costs  of  government  and  contractor 
personnel  supporting  the  test  program  and  the  costs  of  materials  and  supplies, 
petroleum,  lubricants,  and  oil  (PLO);  travel;  and  other  test-specific  requirements  as 
directed  by  the  PMO.  ATC  accounts  for  the  cost  of  each  employee's  service  by 
multiplying  the  actual  hours  worked  by  the  ATC  employee  engaged  in  a  FU  SL  LFT 
task  by  the  specific  wage  rate  for  that  employee's  services.  The  wage  rate  is  computed 
by  adding  the  employee's  hourly  wage  rate  (i.e.,  basic  salary  with  fringe  benefits)  to  the 
hourly  test  support  distributive  rate  identified  for  services  rendered  at  ATC.  The  test 
support  distributive  rate  is  established  by  the  ATC  budget  office  at  the  beginning  of  the 
fiscal  year  by  determining  the  relationship  of  estimated  indirect  costs  of  testing  for  the 
period  (e.g.,  testing  utility  costs,  maintenance  contracts  for  support  vehicles,  hazardous 
waste  disposal,  ammunition  handling,  printing,  range  control,  test  support  supervisors 
above  the  working  leader  level,  costs  of  range  control)  to  projected  direct  labor  hours  to 
be  performed  by  ATC  employees  for  the  period.  As  Department  of  Defense  (DoD) 
customers,  PMOs  do  not  incur  any  additional  charges  for  the  general  and 
administrative  costs  of  ATC  operations. 

The  PMO  pays  for  interruptions  and  shutdowns  that  are  necessitated  by  its  system, 
and  repeated  startup  and  shutdown  cycles  increase  the  costs  incurred  by  the  PMO  over 
and  above  budgeted  amounts.  In  general,  ATC  attempts  to  minimize  the  costs  of  the 
PMO  by  reassigning  those  support  personnel  originally  designated  for  the  FU  SL  LFT  of 
a  system  to  other  jobs  when  shutdowns  occur. 

Vehicle  and  crew  instrumentation  (e.g.,  instrumentation  to  measure  ballistic  shock, 
blast  overpressure,  crew  acceleration,  toxic  gases,  and  thermal  radiation)  costs  include 
only  the  costs  of  labor  associated  with  the  instrumentation  activities,  unless  the 
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instrumentation  is  destroyed  or  damaged  during  the  test  (e.g.,  fire  consumes  a  TV 
camera  in  a  test  article  following  shot).  Expendable  supplies  used  in  testing  (c.g., 
wiring,  connectors,  tubing,  transducers)  are  charged  to  the  PMO.  If  instrumentation 
needed  in  test  execution  is  not  owned  by  ATC,  ATC  investigates  the  feasibility  of 
borrowing  or  renting  that  instrumentation.  If  attempts  to  obtain  instrumentation  by 
other  means  are  not  successful,  ATC  purchases  the  instrumentation  and  charges  the 
cost  to  the  PMO.  Training  costs  that  are  generic  to  instrumentation  for  multiple  systems 
are  budgeted  into  the  test  support  distributive  rates  of  ATC,  but  system-specific 
training  costs  must  be  directly  reimbursed  by  the  PMO  of  the  system. 

ATC  provides  the  PMO  with  a  formal  cost  estimate  prior  to  test  execution  and 
requires  the  PMOs  to  front-forward  the  budgeted  costs  of  ATC  activities  (i.e.,  pay  prior 
to  performance  of  FU  SL  LFT  activities)  and  incrementally  fund  the  program  per  fiscal 
year.  If  the  formal  cost  estimates  funded  by  the  PMO  exceed  actual  costs,  the  PMOs  are 
allowed  to  withdraw  the  unused  funds  or  apply  those  funds  to  a  follow-on  test 
program,  an  upgrade  in  ATC  s  facilities,  or  the  purchase  of  materials  or  instruments  for 

a  future  test  program. 

ATC  reports  ongoing  LFT  program  costs  to  the  PMO  upon  request.  Costs  are 
prepared  with  a  software  package  that  computes  costs  incurred  to  date  on  the  basis  of 
activities  completed  and  ATC  assigned  costs  for  actual  employee  services  rendered  and 
materials  used.  The  liaison  officer  of  the  PMO  has  direct  access  to  the  computer  system 
(i.e.,  TRMS)  for  tracking  costs. 

3.4.3  Cost  of  HQ,  TECOM  Activities.  HQ,  TECOM  oversees  the  coordination  of 
the  DTP  and  the  DTR  of  systems  designated  to  be  tested  by  TECOM  centers  and  is 
responsible  for  the  review  of  the  DTP  and  the  DTR  for  LFTs  assigned  to  non-TECOM 
test  centers.  In  addition,  HQ,  TECOM  personnel  serve  on  the  LFIPT  led  by  EAC.  The 
TECOM  test  manager  assists  the  evaluator  and  the  PMO  in  the  development  of  the 
LFT&E  strategy,  the  resolution  of  LFT&E  issues  such  as  scheduling  and  test 
configuration,  and  the  execution  tasking  to  other  test  centers.  With  the  exception  of 
limited  travel  costs,  costs  incurred  by  HQ,  TECOM  are  personnel  costs  and  are  part  of 
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the  HQ,  TECOM  budget.  These  costs  are  not  reimbursed  by  PMO  funds.  Travel  costs 
related  to  LFT  program  activities  are  reimbursed  by  the  PMO. 

3.4.4  Cost  of  USAOC&S  Activities.  The  major  FU  SL  LFT&E  activity  conducted 
by  USAOC&S  is  BDAR  analysis,  in  addition  to  participation  on  both  the  LFIPT  led  by 
EAC  and  the  damage  assessment  team  led  by  SLAD.  In  an  effort  separate  from  the 
damage  assessment  effort  led  by  SLAD,  USAOC&S  conducts  the  BDAR  analysis  to 
determine  the  ability  of  the  system  tested  to  be  returned  to  an  operational  status  after  a 
hit  from  threat  munitions  (i.e.,  vulnerability  test)  or  the  ability  of  the  threat  vehicles  (i.e., 
targets)  to  be  returned  to  operational  status  after  a  hit  from  the  system  tested  (i.e., 
lethality  test).  Reimbursement  by  the  PMO  to  USAOC&S  for  BDAR  analysis  and 
participation  in  damage  assessment,  activities  generally  performed  by  military 
personnel,  is  made  only  for  travel-related  costs  required  to  accomplish  these  activities. 


3.4.5  Cost  of  EAC  Activities.  The  three  major  LFT  activities  performed  by  EAC  are 
(1)  serving  as  lead  for  the  LFIPT,  working  group  under  the  T&E  IPT;  (2)  preparation  of 
the  EDP,  an  independent  evaluation  of  the  FU  SL  LFT;  and  (3)  preparation  of  the  SER. 
PMOs  are  responsible  for  all  costs  related  to  FU  SL  LFT&E  except  for  the  costs  of  those 
organizations  that  have  mission-related  responsibilities  in  the  LFT&E  program, 
including  EAC  as  a  division  of  OPTEC.  EAC  reports  that  PMOs  are  "taxed"  a  certain 
amount  for  evaluation  activities  performed  for  certain  Acquisition  Category  (ACAT) 
systems.  The  dollar  amount  in  the  PM  budget  line  designated  for  OPTEC  activities  is 
determined  at  the  Pentagon  level  and  is  part  of  the  original  allocation  of  funds  to  the 
PMO.  PM  personnel  report  that  funds  required  by  EAC  and  Operational  Evaluation 
Command  (OEC)  divisions  to  perform  evaluation  activities  relevant  to  their  systems  are 
part  of  the  PMO  budget,  and  funds  are  distributed  to  these  agencies  during  the  fiscal 
year. 

3.5  Benefits  of  FU  SL  LFT&E.  DoD  Regulation  5000.2-R  reports  [9]: 

The  objective  of  LFT&E  is  to  provide  a  timely  and  reasonable  assessment  of 
the  V/L  of  a  system  as  it  progresses  through  its  development  and  prior  to 
full-rate  production.  In  particular: 
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•  to  provide  information  to  decision  makers  on  potential  user  casualties, 
vulnerabilities,  and  lethality,  taking  into  equal  consideration 
susceptibility  to  attack  and  combat  performance  of  the  system; 

•  to  ensure  that  knowledge  of  user  casualties  and  system  vulnerabilities 
or  lethality  is  based  on  testing  of  the  system  under  realistic  combat 
conditions; 

•  to  allow  any  design  deficiency  identified  by  the  testing  and  evaluation 
to  be  corrected  in  design  before  proceeding  beyond 

low-rate  initial  production;  and 

•  to  assess  battle  damage  repair  capabilities  and  issues  ( . . .  not  a  statutory 
requirement  of  LFT&E, ...  assess  such  capabilities  whenever  prudent 
and  affordable). 

Supporters  of  mandated  testing  suggest  there  are  significant  benefits  associated  with 
requiring  FU  SL  LFT&E  for  major  U.S.  weapon  systems.  Discussion  of  the  rationale  of 
proposed  benefits  is  presented  in  the  paragraphs  that  follow. 

•  FU  SL  LFT&E  provides  an  opportunity  to  identify  unknown  vulnerabilities  of 
the  system,  thus  allowing  design  or  engineering  changes  to  be  made  prior  to  the 
system  entering  full-scale  production. 

•  FU  SL  LFT&E  provides  insights  into  BDAR  potential. 

•  FU  SL  LFT&E  provides  an  opportunity  to  validate  M&S  tools  developed  by 
analysis  groups. 

•  FU  SL  LFT&E  provides  valuable  input  to  doctrine,  tactics,  and  training  decisions 
of  system  users. 

•  FU  SL  LFT&E  requirement  helps  to  ensure  adequate  consideration  is  given  to 
V /L  issues  throughout  the  acquisition  cycle. 

3.5.1  FU  SL  LFT&E  and  System  V/L.  The  General  Accounting  Office  (GAO)  reports 
that  FU  SL  LFT&E  provides  the  opportunity  not  only  to  identify  unknown  unknowns  in 
the  tested  system,  but  also  to  gain  insight  into  the  system's  known  unknowns  that  are 
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suspected  but  not  well  understood.  The  GAO  researchers  propose  that  FU  SL  LFT&E  is 
the  only  V/L  assessment  method  that  provides  direct  "visual  observation  of  the 
weapon/ target  interaction  under  realistic  combat  conditions  [10]. 

In  general,  FU  SL  LFT&E  provides  an  opportunity  to  gain  insights  into  (1)  the 
synergistic  effects  of  multiple  damage  mechanisms,  (2)  the  phenomena  of  blast  and 
shock,  (3)  the  effects  of  fire,  toxic  fumes,  and  the  fire  suppression  system,  (4)  the  effects 
of  secondary  and  cascading  damage  (e.g.,  secondary  debris  and  ricochet)  not  revealed 
in  component  or  subsystem  testing,  (5)  the  effects  of  noncritical  component  damage  on 
critical  components,  and  (6)  the  effectiveness  of  redundancy  and  other  vulnerability 
reduction  measures  with  respect  to  continued  operation  of  the  system  tested  [11-14].  In 
addition,  LFT  provides  data  to  support  target  signature  impact  research  and  the 
quantification  of  radiation  hazard  in  target  vehicles. 

Proponents  of  FU  SL  LFT&E  argue  these  insights  can  lead  to  the  correction  of  errors 
and  defects  through  redesign  and  engineering  changes  of  the  weapon  system  at  the 
component,  subsystem,  and  system  levels,  thus  decreasing  the  potential  for  personnel 
casualties,  system  and  environmental  losses  and  increasing  the  probability  that  the 
system  will  be  able  to  complete  its  mission  when  operating  in  a  hostile  environment. 

Mr.  James  F.  O'Bryon,  DDOT&E  (LFT),  believes  LFT&E  motivates  decision-makers  to 
find  feasible  solutions  for  V/L  deficiencies  [11].  He  cites  multiple  fixes  that  have  been 
implemented  as  a  result  of  FU  SL  LFT&E,  including  the  shielding  of  components,  the 
adjustment  of  trigger  thresholds,  and  the  revision  of  stowage  compartments. 

3.5.2  FU  SL  LFT&E  and  BDAR  Activities.  Although  not  a  statutory  requirement  of 
LFT,  FU  SL  LFT&E  provides  valuable  input  to  decisions  relative  to  BDAR  activities.  FU 
SL  LFT&E  affords  insights  into  the  adequacy  of  current  stock  levels  of  spare  parts  and 
the  effectiveness  of  current  BDAR  and  troubleshooting  procedures  [10, 15, 16].  BDAR 
activities  related  to  FU  SL  LFT  produce  experiences  from  which  new  repair  techniques 
and  procedures  are  able  to  be  developed  and  provide  realistic  time  estimates  for 
returning  damaged  vehicles  to  combat. 
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3.5.3  FU  SL  LFT&E  and  M&S.  FU  SL  LFT  data  provide  input  to  the  calibration/ 
validation  of  lethality  and  vulnerability  models.  Because  these  V /L  models  are  used  to 
develop  pre-shot  predictions  for  FU  SL  LFT&E  and  may  be  used  to  extend  the  results  of 
testing,  the  data  provided  by  the  FU  SL  LFT&E  of  any  one  system  may  be  significant  to 
the  FU  SL  LFT&E  of  future  systems.  In  the  past,  critics  of  FU  SL  LFT&E  conceded  that 
tests  may  disclose  valuable  V/L  issues  that  had  been  overlooked  in  existing  M&S  tools 
(e.g.,  toxic  fumes  or  penetrator  break-up  in  BFVS)  but  submitted  that  FU  SL  LFT&E  data 
were  unable  to  provide  information  of  significant  value  in  the  validation  of  models  [10, 
17].  In  recent  years,  however,  FU  SL  LFT&E  data  have  provided  the  bases  for 
validation  activities.  For  example,  the  entire  validation  of  the  MUVES-S2  model  for  the 
Bradley  A3  LFT  was  based  on  the  results  of  the  Bradley  High  Survivability  (HS)  LFT, 
and  the  validation  of  the  SAFE  model  for  the  ATACMS  LFT&E  was  based  on  the  results 
of  the  arena  tests  that  were  part  of  the  LFT  program.  Similar  plans  exist  for  validation 
of  models  to  be  used  in  the  FU  SL  LFT&E  of  the  M1A2  System  Enhancement  Package 
(SEP).  Data  collected  from  the  LFT  activities  of  one  system  may  provide  useful  data  for 
the  V&V  of  models  to  be  used  in  the  T&E  activities  of  similar  systems  or  later  models  of 
the  same  system. 

3.5.4  FU  SL  LFT&E  and  Users'  Decisions.  FU  SL  LFT&E  provides  valuable  input  to 
the  development  of  both  the  procedures  for  preparing  crew  and  system  for  battle  and 
the  doctrine  and  tactics  for  a  system's  operations  in  hostile  environments  [13-15].  Data 
prepared  for  FU  SL  LFT  incidents  furnish  information  for  training  simulators  and 
operational  tests'  assessments  [14, 15]. 

3.5.5  FU  SL  LFT&E  and  Survivability/Lethality  Assessment  Programs.  Because 
the  results  of  testing  cannot  be  the  sole  basis  for  V/L  assessment,  it  is  important  that  an 
appropriate  weight  is  given  to  V/L  issues  in  the  design  and  engineering  phases  of  the 
system's  development  [17].  Supporters  of  FU  SL  LFT&E  suggest  congressionally 
mandated  LFT&E  helps  to  ensure  adequate  attention  is  given  to  V/L  issues  throughout 
the  acquisition  cycle  by  the  PMO  and  contractor(s)  [12, 13]. 

Proponents  of  FU  SL  LFT&E  question  whether  the  discipline  to  proceed  with  a 
well-organized  program  of  V/L  assessment  would  continue  should  FU  SL  LFT&E 
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requirements  be  terminated.  Mr.  Thomas  Julian,  Action  Officer,  DOT&E  (LFT),  reports 
the  results  of  a  recent  survey  of  PMs,  testers,  and  analysts  indicate  the  majority  of  those 
surveyed  believe  "the  old  way  of  doing  business"  would  return  if  FU  SL  LFT&E  were 
no  longer  required  of  U.S  weapon  systems  prior  to  entering  full-scale  production  [15]. 
The  high  visibility  of  the  FU  SL  LFT&E  mandate  appears  to  provide  a  certain  degree  of 
motivation  for  PMOs  and  contractors  to  consider  V/L  issues  over  all  phases  of 
acquisition  [17].  Some  suggest  this  high  visibility  also  helps  to  ensure  a  higher  level  of 
congressional  funding  for  V/L  assessment  activities  than  may  be  provided  for 
alternative  strategies  of  assessment  that  do  not  include  the  FU  SL  LFT  element. 


3.6  Project  Manager  Offices:  Impacts  of  FU  SL  LFT&E.  Visits  to  the  AT ACMS 
PMO,  AMCOM,  Redstone  Arsenal,  AL,  and  the  Abrams  and  Bradley  PMOs,  TACOM, 
Warren,  MI,  were  part  of  the  information-gathering  process  for  this  study.  In  each 
location,  members  of  the  PMO  participated  in  discussions  relevant  to  the  programs  of 
survivability  /lethality  in  operation  for  their  respective  systems  and  the  role  of  FU  SL 
LFT&E  in  assessment  of  system  V/L.  Prior  to  visits  with  PM  personnel,  questions 
concerning  the  system  and  survivability/lethality  assessment  were  sent  to  the  PMOs. 
Discussions  focused  on  the  responses  of  the  PM  personnel  to  those  questions  and  topics 
that  arose  from  their  responses.* 


Discussions  with  PMO  personnel  covered  a  wide  range  of  topics,  including: 
•  objectives  of  the  research  study, 


•  current  methodologies  requiring  the  identification  and  measurement  of  the 
impacts  of  competing  T&E  strategies. 


•  organization  of  the  PMO  and  the  role  of  PMO  in  system  acquisition. 


•  The  views  of  PM  personnel  described  in  this  report  were  not  expressed  by  all  persons  in  all  three  PMOs. 
Although  a  group  of  core  topics  formed  the  foundation  for  all  PMO  discussions,  there  were  some  topics 
explored  in  only  one  or  two  offices.  In  general,  personnel  in  any  one  office  did  not  disagree  with  the 
views  expressed  by  other  persons  in  that  same  office  participating  in  the  discussion.  Following  each 
PMO  visit,  PMO  personnel  were  sent  a  summary  statement  of  the  questions  asked  and  the  responses 
recorded  during  the  visit  and  were  asked  for  comments.  PM  personnel  were  also  given  an  opportunity 
to  review  this  part  of  the  report  (i.e..  Project  Manager  Offices:  Impacts  of  FU  SL  LFT&E)  prior  to 
submission. 
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•  phases  of  acquisition  and  the  activities  relevant  to  each  phase, 

•  history  of  live-fire  testing  of  earlier  models  of  the  identified  system, 

•  contracts  for  all  phases  of  system  development  and  production,  including  use  of 
CATV  and  target  costing  in  contracts, 

•  production  history  of  current/ earlier  models  of  the  system  and  related  costs, 

•  effects  of  foreign  military  sales  of  system  on  contractor  charges  to  government, 

•  responsibilities  of  the  contractor  and  government  agencies  in  developmental  and 
operational  testing, 

•  budgeting  for  FU  SL  LFT&E  and  follow-on  tests, 

•  activities  of  FU  SL  LFT&E  and  the  role  of  the  PMO  in  those  activities, 

•  costs  incurred  in  FU  SL  LFT&E,  reported  and  unreported, 

•  design  of  the  FU  SL  LFT&E, 

•  role  of  M&S  in  V/L  assessment,  and 

•  benefits  of  FU  SL  LFT&E  or  lessons  learned  from  FU  SL  LFT&E. 

Much  of  the  specific  system  information  gathered  from  the  PM  discussions  has  been 
incorporated  into  other  sections  of  this  report.  This  part  of  the  report  focuses  on  the 
perceptions  of  PM  personnel  relative  to  the  costs  and  benefits  of  FU  SL  LFT&E  as  one 
part  of  the  survivability /lethality  assessment  process  and  the  suggestions  of  PM 
personnel  for  improving  the  cost-effectiveness  of  the  assessment  process. 

Discussions  related  to  the  impacts  of  FU  SL  LFT&E  emphasized  four  main  topics: 

(1)  the  objectives  of  FU  SL  LFT&E,  (2)  the  effect  of  the  system's  existing  program  of 
survivability /lethality  evaluation  on  the  design  of  the  FU  SL  LFT,  (3)  the  alternatives  to 
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FU  SL  LFT&E,  including  M&S,  and  (4)  the  lessons  learned  from  a  completed  FU  SL 
LFT&E. 

3.6.1  Objectives  ofFU  SL  LFT&E.  Guidelines  for  LFT&E  cite  the  importance  of 
carrying  out  tests  sufficiently  early  in  the  development  phase  to  allow  for  design 
deficiencies  to  be  detected  and  corrected  in  the  weapon  system  tested.  Objectives  of 
LFT&E  include  (1)  ensuring  that  knowledge  of  user  casualties  and  system 
vulnerabilities  and  lethality  is  based  on  realistic  testing  of  the  system  configured  for 
combat  against  expected  threats  and  (2)  gaining  insights  into  potential  design  flaws  so 
that  they  can  be  corrected  before  the  system  enters  full-rate  production  [9]. 

In  discussions  with  PM  personnel,  questions  were  raised  as  to  whether  FU  SL 
LFT&E,  conducted  at  the  conclusion  of  the  Engineering  and  Manufacturing 
Development  (EMD)  phase,  occurred  too  late  in  the  acquisition  process  to  impact 
design  or  secure  (relatively)  inexpensive  fixes.  The  ATACMS  PMO  argued  that  if  a 
discovery  were  made  in  a  FU  SL  LFT  relative  to  the  adequacy  of  the  system  s  bomblet 
lethality,  subsequent  changes  in  the  bomblet  would  affect  the  entire  production  line. 
Changes  in  design  made  late  in  the  production  cycle  would  impact  all  qualitative  tests 
completed  prior  to  the  design  change  and,  in  turn,  would  impact  the  system  evaluation 
required  prior  to  the  system  entering  full-scale  production.  PM  personnel  emphasized 
that  problems  related  to  design  and  engineering  requirements  need  to  be  forecasted  as 
early  in  the  acquisition  process  as  possible.  FU  SL  LFT  is  often  too  late  to  afford 
effective  fixes  in  a  cost-efficient  manner. 

3.6.2  Design  ofFU  SL  LFT&E.  Does  the  system's  existing  program  of 
survivability /lethality  affect  the  plan  of  the  FU  SL  LFT&E?  Is  the  scope  of  the  FU  SL 
LFT  plan  more  restricted  (i.e.,  less  shots)  for  a  system  that  has  a  well-developed  agenda 
for  evaluating  system  survivability/ lethality  built  into  its  acquisition  program  than  the 
scope  of  the  FU  SL  LFT  plan  for  systems  without  such  an  agenda?  Is  the  FU  SL  LFT  of  a 
system  with  a  building-block  program  of  survivability  redundant  (i.e.,  a  series  of  tests 
repeated  at  an  independent  location)? 
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Abrams  PMOs  believe  the  well-managed  survivability  test  program  associated  with 
Abrams  systems  addressed  most  of  the  vulnerability  issues  prior  to  the  M1A2  FU  SL 
LFT&E.  The  Ml  A2  FU  SL  LFT&E  in  1993  confirmed  the  PMO  conviction  that  the  Ml  A2 
was  a  very  survivable  tank  based  on  the  limitations  of  its  69.9-ton  gross  weight.  The 
Abrams'  PMO  reports  that  from  its  initial  concept,  the  Abrams  was  developed  with 
survivability  as  its  first  priority,  and  the  results  of  the  1993  FU  SL  LFT  came  as  no 
surprise  to  the  PMO.  In  its  different  variants,  the  Abrams  has  undergone  extensive 
survivability  testing,  including  tests  of  its  armor,  structures,  and  ballistic  hull  and  turret 
(BH&T),  as  well  as  system-level  tests  not  under  independent  congressional  LFT&E 
auspices.  PM  personnel  questioned  whether  congressionally  mandated  FU  SL  LFT&E 
was  not  a  very  expensive  final  check  on  crew  and  tank  survivability  under  live-fire 
conditions.  Several  interviewees  voiced  the  belief  that  when  EMD  includes  a  well- 
designed  and  well-managed  survivability /vulnerability  program,  the  results  of  FU  SL 
LFT&E  are  predictable  and  the  likelihood  of  major  surprise  outcomes  is  small. 

3.6.3  Alternatives  to  FU  SL  LFT&E.  Individuals  interviewed  acknowledge  that  a 
LFT  program  provides  insight  into  the  areas  of  component,  subsystem,  and  system 
vulnerability/lethality;  damage  and  casualty  mechanisms;  perforating  and 
nonperforating  impacts  on  the  system;  and  system-  or  threat-specific  subissues  such  as 
compartmentalization,  stowed  munitions,  fuel  and  hydraulics,  fire  suppression 
systems,  and  blast,  shock,  and  incendiary  effects.  The  question,  however,  remains, 

"Are  there  more  economical  ways  than  FU  SL  LFT&E  to  obtain  the  information  needed 
by  decision-makers  or  to  discover  the  unknown  vulnerabilities?"  Is  FU  SL  LFT&E 
designed  with  the  objective  of  obtaining  information  needed  by  decision-makers  that  is 
not  available  from  other  sources? 

Abrams  PM  personnel  provided  an  example  in  which  off-line  tests  were  conducted 
to  verify  the  design  of  two  very  important  survivability  fixes.  The  tests  consisted  of 
seven  high-caliber  shots  on  an  earlier  model  with  fixes  incorporated.  The  total  test  cost 
was  reported  to  be  approximately  25%  of  the  cost  of  one  FU  SL  LFT&E  shot.  The 
question  then  becomes,  "If  resources  are  limited,  is  adequate  consideration  given  to 
alternative  methods  (i.e.,  methods  other  than  FU  SL  LFT&E)  to  gather  information 
required  for  assessment  decisions?"  Interviewees  agreed  that  the  response  to  this 
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question  is  different  for  each  FU  SL  LFT&E.  Citing  recent  technological  developments, 
Abrams  PM  and  Bradley  interviewees  noted  that  in  the  survivability  programs  of 
earlier  models,  some  system-level  effects  were  only  testable  in  FU  SL  LFT&E.  Currently, 
however,  some  of  the  same  effects  can  be  assessed  in  off-line  tests  (e.g.,  effects  of  blast 
and  shock  in  WSMR  air  blast  tests). 


A  second  issue  raised  in  discussions  concerned  the  balance  of  engineering-shot  lines 
(i.e.,  shots  selected  to  address  specific  V/L  issues)  and  random-shot  lines  (i.e.,  shots 
generated  from  likely  hit  points)  in  the  FU  SL  LFT  program.  Interviewees  suggested 
that  controlled  damage  tests  (CDTs)  and  the  engineered  shots  of  FU  SL  LFT&E  often 
produce  the  data  that  decision-makers  require  with  minimum  test  asset  repairs. 
Personnel  expressed  frustration  that  in  some  situations  the  PMO  is  aware  of  and 
concedes  damage  with  planned  random  shots,  but  the  random  shots  remain  part  of  the 
FU  SL  LFT&E  plan.  Frustration  was  also  expressed  that  attempts  by  the  PMO  to 
modify  configurations  of  the  test  items  to  eliminate  expensive  line  replacement  units 
were  often  unsuccessful.  Interviewees  hastened  to  add  that  they  understood  random 
shot  lines  were  a  part  of  the  realistic  combat  setting  required  in  FU  SL  LFT  and  that  a 
balance  of  random  and  engineering  shots  was  important  to  the  perception  of  a  fair  test 
of  the  system's  V/L.  Concern  was  expressed,  however,  that  many  more  dollars  are  now 
spent  on  random  shots  than  on  engineering  shots  in  which  more  useful  information  is 
often  received. 

In  particular,  PM  personnel  were  asked  if  they  believed  M&S  could  serve  as  an 
effective  substitute  for  live-fire  testing  in  the  assessment  of  the  V/L  of  a  weapon  system. 
Interviewees  were  in  agreement  that  both  M&S  and  testing  are  needed  in  V/L 
assessment — that  they  serve  as  complements  to  rather  than  substitutes  for  each  other. 

They  emphasized  that  testing  is  required  in  the  development  of  M&S  tools  to  ensure 
M&S  robustness  and  fidelity;  in  turn,  M&S  is  needed  to  extend  the  results  of  testing  to  a 
broad  range  of  weapon  system-target  encounters. 

Although  all  interviewees  expressed  the  opinion  that  significant  progress  in  M&S  is 
needed  before  any  substantial  cuts  in  testing  would  be  feasible,  interviewees  expressed 
the  hope  that  M&S  would  eventually  be  mature  enough  to  allow  substantial  reduction 
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in  the  required  number  of  FU  SL  LFT  shots.'  They  cited  the  important  role  that  M&S 
plays  in  the  design  of  the  system  (e.g.,  layouts  of  interior  or  placement  of  armor)  and 
the  advancements  in  modeling  of  damage  mechanisms  (e.g.,  penetrating  shots).  They 
strongly  encourage  testers  and  modelers  to  continue  to  work  together  to  ensure  that 
testers  understand  the  type  of  information  that  is  required  for  building  models. 
ATACMS  PMO  cited  the  valuable  input  provided  by  the  tire  tests  (i.e.,  tests  related  to 
the  ATACMS  Block  IA  LFT)  to  the  subsequent  modeling  of  tire  effects.  In  a  similar 
manner,  ATACMS  PM  personnel  believed  that  ATACMS  follow-on  tests  would 
contribute  to  the  modeling  of  fire  effects. 

Although  all  groups  supported  the  effort  to  devote  more  dollars  to  the  development 
and  improvement  of  M&S  tools,  all  were  quick  to  point  out  that  the  funds  available  for 
testing  and  M&S  are  limited  and  that  tradeoffs  are  a  fact  of  life.  Several  interviewees 
suggested  more  effort  should  be  made  to  integrate  modeling  and  testing  to  provide 
information  needed  for  decision-makers  in  a  more  cost-effective  manner.  Views  were 
expressed  that  even  with  its  current  limitations,  M&S  can  be  combined  with 
nondestructive  testing  to  produce  valuable  data.  All  realized  that  this  is  a  complex 
situation,  as  funding  of  additional  tests  and  experiments  to  improve  the  fidelity  of 
models  often  leads  to  more  questions  that  must  be  resolved  by  additional  testing. 

In  general,  there  appeared  to  be  consensus  across  the  groups  that  two  major  issues 
related  to  M&S  remain  to  be  resolved:  (1)  how  to  obtain  the  financial  resources  needed 
for  the  development  of  high  accuracy  M&S  tools  and  (2)  how  to  ensure  the  acceptability 
of  an  increased  role  for  M&S  in  the  assessment  of  survivability/lethality  of  a  weapon 
system. 

3.6.4  Lessons  Learned  from  FU  SL  LFT&E.  Although  it  is  difficult  to  measure  the 
benefits  of  FU  SL  LFT  ex  ante ,  it  may  be  helpful  to  look  at  the  lessons  learned  from  a 

‘  Some  argue  that  a  substantial  reduction  in  the  required  number  of  shots  has  already  taken  place  for  a 
variety  of  reasons.  For  example.  Phase  II  of  the  FU  SL  LFT  of  the  Bradley  M2  and  M3  in  1986-1987 
included  over  80  shots  (i.e.,  random,  prescribed,  and  conceded/repeated  shots,  as  well  as  repeated 
Phase  I  and  Development  Test  shots),  as  compared  to  the  scheduled  18  shots  of  the  1998-1999  Bradley 
A3  FU  SL  LFT. 
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completed  FU  SL  LFT&E  program  as  an  ex  post  measure  of  the  effectiveness  of  the  FU 
SL  LFT  activity.  Interviewees  in  the  PMOs  of  the  two  systems  in  this  study  that  had 
completed  a  FU  SL  LFT&E  were  asked  about  their  perceptions  of  the  benefits  of  the  test 
and  what  actions  were  taken  in  response  to  system  deficiencies  discovered  in  the  test 
series. 

ATACMS  PM  personnel  reported  that,  while  the  FU  SL  LFT  for  the  ATACMS  Block 
LA  did  not  answer  all  questions  about  the  effectiveness  of  all  M74  bomblet  damage 
mechanisms,  it  did  point  out  the  potential  for  product  improvement.  It  also  pointed  out 
the  need  for  more  modeling  and  analysis  in  the  characterization  of  the  bomblet  earlier 
in  the  acquisition  process.  At  the  time  of  the  interview,  the  ATACMS  PM  office  was 
planning  follow-on  tests  (required  before  full-scale  production  could  begin)  to  further 
examine  the  effectiveness  of  the  M74  bomblet  against  potential  target  elements. 

Abrams  PM  personnel  reported  that  the  M1A2  Abrams  FU  SL  LFT  verified  some 
fixes  on  problem  areas  discovered  on  prior  FU  SL  LFTs  and  indicated  some  new 
vulnerabilities  that  could  be  eliminated  or  minimized  with  modifications.  Problem 
areas  that  surfaced  were  reported  to  be  small.  Personnel  pointed  out  that  two  prior 
fixes  had  been  verified  on  off-line  tests  related  to  LFT&E  and  that  FU  SL  LFT  basically 
confirmed  what  the  PMO  already  knew  about  the  Ml  A2.  They  believe  the  results  of  the 
FU  SL  LFT  support  their  belief  that  the  Abrams  survivability  test  program  previously 
addressed  most  of  the  vulnerability  issues  of  the  weapon  system.  Personnel  estimated 
that  80-90%  of  data  uncovered  in  1993  FU  SL  LFT  could  now  be  discovered  in  a  test 
program  that  did  not  include  FU  SL  LFT.  Personnel  acknowledge,  however,  that  some 
of  the  technology  that  is  available  at  this  date  was  not  available  at  the  time  of  the  Ml  A2 
Abrams  FU  SL  LFT.  For  instance,  the  use  of  shock  simulators  currently  affords  a 
method  of  testing  off-line  that  was  unavailable  in  1993.* 


The  ballistic  shock  simulator  developed  by  ATC  is  not  a  replacement  for  shock  testing  on  a  full-up 
vehicle.  Although  useful  in  identifying  weak  components  prior  to  FU  SL  LFT,  the  simulator  offers  no 
guarantee  that  a  component  that  does  not  fail  on  the  ballistic  shock  simulator  will  survive  the  FU  SL 
LFT. 
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Abrams  personnel  reported  that  potential  fixes  are  briefed  as  part  of  MS  m 
deliberations  but  that  modifications  or  fixes  are  determined  not  solely  by  the  PMO  but 
by  an  oversight  committee  with  input  from  system  users  and  trainers,  system 
developers,  and  logistics  personnel.  Decisions  relative  to  which  fixes  are  implemented 
are  made  on  the  bases  of  the  priorities  assigned  fixes  uncovered  in  testing  and  the 
availability  of  funds.  Often,  corrective  actions  are  incorporated  into  later  designs  of  the 

system. 

The  Abrams  Ml  A2  FU  SL  LFT&E  resulted  in  12  recommendations,  including  9  fixes 
that  were  initiated  for  production  retrofit  or  incorporation  into  future  tank  design, 

2  that  were  reviewed  but  not  implemented,  and  1  that  was  not  addressed.  In  addition, 
some  LFT  insights  were  incorporated  into  the  tactics,  training,  and  planning  of  the 
system  user  to  improve  the  survivability  of  system  personnel  in  the  combat 
environment. 

3.6.5  Suggested  Actions.  In  discussing  the  effectiveness  of  FU  SL  LFT&E  and  the 
lessons  learned  from  tests  completed,  PMO  personnel  offered  the  following  suggestions 
to  improve  the  cost-effectiveness  of  survivability /lethality  assessment  and,  in 
particular,  the  cost-effectiveness  of  FU  SL  LFT&E  as  part  of  the  assessment  process. 

•  Enlist  multiple  supporters  for  the  development  and  V&V  of  M&S  tools  (OSD, 
PMO,  etc.).  Encourage  modelers  and  testers  to  integrate  efforts. 

•  Provide  additional  funds  in  the  early  phases  of  acquisition  to  allow  up-front 
discussions  of  M&S  with  SLAD  personnel;  currently,  PMOs  are  unable  to  fund 
this  activity  prior  to  the  establishment  of  user  requirements. 

•  Organize  a  round-table  discussion  in  the  early  stages  of  the  acquisition  process 
for  ARL  personnel  and  all  parties  of  T&E  activities  to  consider  carefully 
alternative  methods  of  obtaining  information  needed  by  decision-makers  in  the 
assessment  of  system  V/L. 
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•  Enlist  involvement  of  the  evaluator  very  early  in  the  T&E  process,  using  a  team 
approach  with  the  evaluator  as  a  member  of  the  team.  PM  personnel  suggest  the 
oversight  of  government  evaluators  in  contractor  testing  allows  the  positive 
results  of  those  tests  to  be  included  in  the  government  evaluation.  PM  personnel 
noted  that  this  cooperation  has  been  more  common  in  recent  LFT  programs  than 
in  prior  test  programs. 

•  Consider  limiting  the  scope  of  the  FU  SL  LFT&E  of  those  systems  managed  by 
PMOs  with  well-established  building-block  programs  of  survivability /lethality 
assessment  in  place.  Scope  limitation  could  serve  as  an  incentive  for 
incorporating  V/L  assessment  into  all  phases  of  the  acquisition  cycle. 

•  Consider  carefully  the  balance  of  engineered  and  random  shots  in  FU  SL  LFT, 
weighing  the  cost  incurred  for  information  obtained.  Consider  alternative  tests 
or  forms  of  analysis  and  the  costs  associated  with  those  tests  and  analyses  in 
designing  the  number  and  types  of  shots  in  the  FU  SL  LFT&E  strategy. 

•  Consider  granting  concessions  in  FU  SL  LFT&E.  In  years  prior  to  the 
congressional  LFT&E  mandate,  critical  components  of  a  system  often  were 
removed  before  testing  in  FU  SL  LFT,  if  the  PM  office  conceded  the  system 
would  fail  with  hit.  Some  PM  personnel  questioned  the  cost-efficiency  of  FU  SL 
LFT  congressional  test  requirements  that  mandate  that  critical  components 
remain  in  the  test  asset. 

PMO  personnel  explained  that  some  of  their  problems  with  FU  SL  LFT&E  originate 
with  the  fact  that  PMOs  focus  on  demonstrating  that  a  system  is  in  compliance  with  the 
ORD  for  the  system,  but  FU  SL  LFT&E  often  is  structured  to  evaluate  a  system  beyond 
its  requirements.  Therefore,  PMOs  frequently  find  it  difficult  to  fully  support  LFT&E 
and  the  incorporation  of  design  fixes  that  the  FU  SL  LFT  indicates  are  needed  for  the 
system  that  has  been  evaluated  beyond  its  requirements. 

In  conclusion,  it  is  important  to  emphasize  that  the  PMOs  visited  did  not  discount 
the  benefits  of  an  LFT&E  program.  Interviewees  stated  that  changes  in  materials  and 
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technology  in  recently  produced  weapon  systems  require  a  program  of  LFT&E  and  that 
alternative  tests  may  not  catch  everything  found  in  a  FU  SL  LFT.  Most  of  the  above 
suggestions  offered  by  PM  personnel  attempt  to  address  the  question  of  how  limited 
resources  may  be  employed  to  design  the  most  effective  plan  for  assessing  the 
survivability/lethality  of  a  weapon  system. 

4.  Framework  for  Vulnerability/Lethality  Assessment 

In  the  current  era  of  constrained  resources,  the  effectiveness  and  efficiency  of  a  V/L 
assessment  strategy  may  be  increased  by  the  identification  of  the  specific  set  of  data 
required  for  assessment,  the  determination  of  the  subset  of  the  required  data  currently 
available  from  reliable  sources,  and  the  selection  of  a  cost-effective  assessment  plan  (i.e., 
M&S,  experimentation,  FU  SL  LFT&E,  etc.)  to  fill  the  data  voids.  Although  the 
approach  described  in  the  following  paragraphs  is  applicable  equally  to  lethality 
assessment,  the  discussion  offered  in  this  part  of  the  report  is  limited  to  vulnerability 
assessment. 

In  assessing  the  vulnerability  of  a  weapon  system,  analysts  evaluate  the  system's 
likelihood  to  sustain  personnel  casualties,  suffer  catastrophic  losses,  or  fail  to  complete 
its  mission  when  faced  with  those  threats  deemed  likely  to  be  encountered  in  combat. 
Faced  with  decreased  defense  budgets,  it  is  important  for  analysts  to  employ  an 
assessment  strategy  that  allows  vulnerabilities  to  be  detected  as  early  in  the  acquisition 
process  as  possible,  thus  affording  the  maximum  time  to  develop  fixes  prior  to  the 
system  entering  full-scale  production.  An  effective  vulnerability  assessment  program 
begun  early  in  the  acquisition  process  has  the  potential  to  detect  system  vulnerabilities 
and  reduce  or  minimize  not  only  the  costs  associated  with  combat  losses  but  also  the 
costs  associated  with  retrofit  of  systems  in  the  late  stages  of  production. 


*  Fixes  associated  with  vulnerabilities  detected  late  in  the  acquisition  cycle  are  less  likely  to  be  able  to  be 
achieved  through  redesign  or  re-engineering  of  the  system  and  more  likely  to  be  associated  with  the 
higher  costs  of  retrofitting  of  completed  systems.  Serious  vulnerabilities  detected  late  in  the  acquisition 
cycle  could  lead  to  system  abandonment.  Although  it  is  acknowledged  that  not  all  vulnerabilities  can 
be  eliminated,  knowledge  of  a  system's  vulnerabilities  may  affect  users'  tactics  and  strategies  in  combat. 
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Limited  resources  have  led  to  an  increased  emphasis  on  the  costs  as  well  as  the 
benefits  of  conducting  vulnerability  assessments  of  weapon  systems.  It  is  proposed  m 
this  section  of  the  report  that  an  effective  vulnerability  assessment  strategy  identifies  the 
specific  set  of  data  required  for  vulnerability  assessment,  determines  the  subset  of  the 
required  data  currently  available  from  reliable  sources,  and  then  designs  a  vulnerability 
assessment  plan  (i.e.,  M&S,  experimentation,  FU  SL  LFT&E,  etc.)  to  fill  the  data  voids. 
Alternative  vulnerability  assessment  plans,  designed  to  provide  the  missing  critical 
data,  must  be  compared  on  the  bases  of  (1)  the  capacity  of  each  plan  to  provide  the 
critical  data  (i.e.,  benefits  of  plan),  (2)  the  cost  to  accomplish  each  plan,  and  (3)  the 
uncertainty  (risk)  associated  with  each  plan's  capacity  to  provide  the  critical  data  at  the 
identified  cost  within  the  time  constraints  of  the  assessment  process. 

An  effective  vulnerability  assessment  strategy  includes  both  an  analysis  of  the  likely 
combat  scenarios  and  a  determination  of  the  links  between  the  measures  of  battlefield 
effectiveness  and  the  measures  of  capability  required  of  the  system  to  complete  its 
mission.  In  turn,  the  vulnerability  assessment  plan  is  designed  to  fill  the  data  voids  that 
exist  in  the  analyses  of  (1)  the  relationship  between  the  threat-system  interaction  and 
the  collection  of  components  associated  with  system  and  personnel  vulnerability  that 
have  been  damaged  and  (2)  the  relationship  between  this  collection  of  damaged 
components  and  the  measures  of  capability  required  by  the  system  to  complete  its 
mission. 

The  V/L  Taxonomy  is  suggested  as  the  appropriate  framework  for  identifying  the 
information  required  in  the  vulnerability  assessment  process,  as  well  as  the  data  voids 
to  be  addressed  in  the  vulnerability  assessment  plan  [2].  A  description  of  the 
methodology  for  the  design  of  cost-effective  vulnerability  assessment  plans  and  the 
selection  of  the  optimal  plan  from  those  alternatives  follows  a  review  of  the  Taxonomy. 

4.1  Taxonomy  of  the  V/L  Analysis  Process.  The  V/L  Taxonomy  provides  a 
framework  for  examining  the  elements  of  the  complex  vulnerability  assessment  process. 


'  Time  constraints  must  consider  the  availability  of  hardware  required  to  complete  test  plan  components. 
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The  V/L  Taxonomy  was  first  introduced  by  Deitz  and  Ozolins  in  1989  as  a  conceptual 
framework  for  armored  fighting  vehicle  vulnerability  assessment  in  association  with  the 
computer  simulation  of  the  Abrams  Live-Fire  Field  Testing  [2].  The  general  description 
of  the  V/L  Taxonomy  that  follows  is  based  on  the  dted  publication  plus  several 
subsequent  publications  that  have  discussed  the  analytical  properties  of  the  V/L 
Taxonomy  [2, 18-24]. 

The  V/L  Taxonomy  reflects  the  vulnerability  analysis  process  as  a  sequence  of 
several  levels,  each  containing  specific  information  about  the  analysis  process.  The 
information  in  one  level  is  mapped  to  the  information  in  the  subsequent  level  by  a 
transformation  or  a  mapping  process.  A  description  of  the  taxonomy  includes  four 
levels  with  associated  spaces*  as  shown  in  the  graphical  representation  found  in 
Figure  1/  Mapping  from  higher  to  lower  levels  (i.e..  Level  1  to  Level  2;  Level  2  to 
Level  3;  and  Level  3  to  Level  4),  the  taxonomy  can  be  described  as  follows: 

•  Level  1  defines  the  initial  configuration  of  the  specific  threat  and  weapon 
system  (i.e.,  target  for  which  vulnerability  is  assessed)  encounter.  Space  1 
includes  the  set  (i.e.,  all  combinations)  of  the  initial  configurations  of  the  threat 
and  system  just  prior  to  threat-system  interaction. 

•  Level  2  defines  the  physical  state  or  the  damage  state  of  the  weapon  system  that 
results  from  the  interaction  of  the  threat  and  system.  Space  2  includes  the  set 
(i.e.,  all  combinations)  of  damage  states.  Each  state,  represented  by  a  point  in 
Space  2,  is  generally  expressed  as  an  n-tuple  with  elements  describing  the  status 
of  certain  specific  n  number  of  components  of  interest.  In  the  V/L  Taxonomy, 

*  The  V/L  Taxonomy  also  applies  to  lethality  analysis.  In  this  report,  however,  only  vulnerability 
analysis  is  addressed. 

+  A  level  indudes  all  the  information  necessary  to  define  the  state  of  the  system  at  the  associated  stage  of 
the  vulnerability  analysis.  A  space  of  points  may  be  defined  at  each  level  to  represent  the  state  of  the 
system  at  that  level.  Each  point  in  a  defined  space  is  a  vector  with  a  specific  number  of  elements,  each 
element  referring  to  the  status  of  an  entity  related  to  the  system  (e.g.,  system  components  in  Level  2). 
There  may,  however,  be  multiple  spaces  at  each  level,  each  space  defined  by  a  different  set  of  elements 
in  the  vectors  that  make  up  that  space  at  that  particular  level  [23]. 

t  Figure  1  is  adapted  from  Figure  1:  V/L  Taxonomy  Illustrated  via  a  Mapping  Abstraction,  as  presented 
in  Deitz  and  Starks  [19]. 
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Levels-Descriptors  Transformations 
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Figure  1.  Graphical  Representation  of  the  Taxonomy. 


each  Level  2  state  is  expressed  as  a  damage  state  vector  with  each  element  of  the 
vector  representing  the  status  of  a  component  following  the  threat-system 
interaction. 

The  transformation  of  the  undamaged  system  of  Level  1  to  the  damaged  system 
in  Level  2  (01/2)  is  a  physical  process  and  is  characterized  by  physical  damage 
mechanisms,  such  as  threat  penetrators,  fragments,  shock,  or  fire.  The  process  of 
mapping  a  point  in  Space  1  to  a  point  in  Space  2  is  generally  stochastic,  and  a 
single  point  in  Space  1,  representing  a  threat-system  configuration,  may  map  to 
multiple  points  in  Space  2  with  each  point  in  Space  2  represented  by  a  different 
damage  state  vector.  Thus,  identical  shots  by  a  specific  threat  against  a  specific 
system,  when  repeated,  often  result  in  different  damage  state  vectors  for  the 
targeted  system.  It  is  important  to  note  that  multiple  points  in  Space  1  may  map 
to  the  same  single  damage  state  vector  in  Space  2  [19, 20]. 

•  Level  3  defines  the  capability  state  of  the  weapon  system  that  results  from  the 
threat-system  interaction.  Space  3  includes  the  set  (i.e.,  all  combinations)  of 
capability  states.  Similar  to  the  damage  state  of  Level  2,  the  capability  state  of 
Level  3  is  expressed  often  as  an  m-tuple  with  elements  describing  the  status  of 
certain  specific  m  number  of  capability  or  performance  measures  (e.g., 
capabilities  related  to  firepower,  mobility,  communications,  etc.)  relevant  to  the 
system's  mission.’  Each  state  represents  a  point  in  Space  3  and  is  expressed  as  a 
capability  state  vector  with  elements  representing  specific  levels  of  identified 
capabilities  of  the  system.  The  capability  state  vector  represents  the  aggregate 
capability  of  the  system  to  function  in  areas  relevant  to  the  system's  mission 
following  threat-system  interaction. 

The  transformation  of  a  damage  state  vector  in  Space  2  to  a  capability  state  vector 
in  Space  3  (02(3)  is  a  mapping  of  damaged  components  to  the  resulting  reduced 
capability  state.  It  is  important  to  note  that  although  a  single  damage  state  vector 


In  this  report,  capability  (not  performance)  measures  are  used  to  describe  the  metrics  of  Level  3  of  the 
V/L  Taxonomy. 
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in  Space  2  maps  to  a  single  capability  state  vector  in  Space  3,  multiple  damage 
state  vectors  in  Space  2  may  map  to  the  same  single  capability  state  vector  in 
Space  3. 

•  Level  4  defines  the  operational  utility  state  of  the  weapon  system  that  results 
from  the  aggregate  system  capabilities  measured  in  Level  3.  Data  from  Levels  2 
and  3,  combined  with  information  relative  to  the  combat  scenario  (i.e.,  threat 
environment,  physical  environment,  and  mission  objectives)  and  users'  tactics 
and  doctrine,  are  used  to  determine  the  mission  status  points  or  the  measures  of 
effectiveness  (MOEs)  included  in  Space  4. 

The  transformation  of  the  capability  vector  in  Space  3  to  the  operational  utility 
state  in  Space  4  (03/4)  is  often  perceived  as  outside  of  the  process  of  vulnerability 
analysis  and  requiring  the  input  of  system  users  and  military  strategists.  In 
reality,  this  mapping  is  often  included  as  part  of  the  vulnerability  assessment 
with  input  provided  by  the  system  user  or  strategist. 

4.2  V/L  Taxonomy  and  Vulnerability  Assessment.  The  V/L  Taxonomy  presented 
in  the  previous  section  provides  a  framework  for  an  analysis  of  the  vulnerability 
assessment  process.  In  the  vulnerability  assessment  of  a  weapon  system,  analysts  ask 
the  question,  "To  what  extent  will  the  interactions  of  the  weapon  system  and  the  threats 
that  the  system  is  likely  to  encounter  in  combat  result  in  personnel  casualties  (i.e., 
personnel  vulnerability)  or  the  inability  to  complete  the  system's  missions  (i.e.,  system 
vulnerability)?" 

Before  analysts  are  able  to  design  cost-effective  vulnerability  assessment  plans  to 
address  the  aforementioned  question,  however,  they  must  ascertain  what  information  is 
needed  to  adequately  answer  the  assessment  question(s)  and  determine  the  subset  of 
that  information  that  is  available  from  reliable  sources.  A  comparison  of  the  required 
information  to  the  subset  of  available,  relevant,  and  reliable  information  leads  to  the 
identification  of  the  data  voids  (i.e.,  subset  of  the  required  information  that  is 
unavailable  or  unable  to  be  relied  upon).  The  data  voids  define  the  information  that 
must  be  gathered  in  the  vulnerability  assessment  plan. 
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4.2.1  The  Required  Data.  To  determine  what  data  are  needed  to  evaluate  the  crew 
and  system  vulnerability  to  threats  likely  to  be  encountered  in  combat,  the  analysts 
must  have  a  clear  definition  of  personnel  and  system  vulnerability.  U.S.  Defense 
acquisition  policies  define  vulnerability  as  [9]: 

The  characteristic  of  a  system  that  causes  it  to  suffer  a  definite  degradation 
(loss  or  reduction  of  capability  to  perform  its  designated  mission)  as  a  result  of 
having  been  subjected  to  a  certain  (defined)  level  of  effects  in  an  unnatural 
(man-made)  hostile  environment. 

In  vulnerability  assessment,  analysts  are  interested  not  only  in  threat-system 
interactions  that  render  the  crew  unable  to  complete  its  mission  but  also  in  interactions 
that  result  in  injuries  to  the  system's  crew  members  and  passengers  even  though  the 
mission  is  able  to  be  completed.  System  vulnerability  includes  any  damage  to  the 
weapon  system,  including  the  catastrophic  loss  of  the  system,  that  does  not  allow  for 
mission  completion.  To  provide  decision-makers  with  data  characterizing  the 
vulnerability  of  the  system  and  its  personnel,  analysts  must  be  able  to  collect 
information  relevant  to  the  following  questions: 

1.  What  are  the  likely  combat  scenarios  in  which  this  system  will  operate? 

2.  Considering  the  characteristics  of  the  combat  scenario  (i.e.,  threat,  physical 
environment,  and  mission  objectives)  and  users'  tactics  and  doctrine,  what 
activities  will  the  system  be  required  to  perform? 

3.  What  enables  mission  success  or  completion  in  the  identified  combat  scenarios? 
How  is  battlefield  effectiveness  of  the  system  or  mission  success  measured? 


Traditionally,  question  nos.  1, 2,  and  3  have  been  considered  to  be  beyond  the  purview  of  the 
vulnerability  analyst.  The  methodology  described  in  this  report  emphasizes  the  importance  of 
understanding  the  relationship  between  the  tasks  that  must  be  completed  to  ensure  a  successful  mission 
and  the  minimum  levels  of  system  capabilities  needed  to  complete  those  tasks  in  a  cost-effective 
vulnerability  assessment. 
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4.  What  specific  capabilities  (i.e.,  capabilities  in  areas  of  mobility,  firepower, 
communications,  etc.)  must  be  evaluated  in  the  assessment  of  the  effectiveness  of 
the  system? 

5.  What  levels  of  the  specific  capabilities  identified  in  #4  must  exist  to  ensure 
mission  completion? 

6.  What  subsystems  of  the  weapon  system  must  be  in  operation  to  produce  the 
levels  of  capabilities  identified  in  #5  as  necessary  for  mission  completion? 

7.  What  are  the  critical  components  of  the  subsystems  identified  in  #6?  * 

8.  What  type  of  damage  (i.e.,  personnel  casualties,  catastrophic  loss  of  the  system, 
damage  to  critical  components  of  the  system)  to  the  weapon  system  is  expected 
from  the  interaction  of  the  system  and  threats  likely  to  be  encountered  in 
combat? 

9.  Given  the  damage  states  identified  in  #8  and  the  critical  components  of  the 
system  identified  in  #7,  to  what  extent  are  the  system  and  its  personnel 
vulnerable? 

4.2.2  The  Taxonomy  of  Vulnerability  Assessment.  A  few  changes  to  the 
V /L  Taxonomy  shown  in  Figure  1  produce  the  Taxonomy  of  Vulnerability  Assessment 
(Taxonomy  VA)  in  Figure  2,  an  appropriate  framework  for  addressing  the  preceding 
questions.1  The  Taxonomy  VA  provides  the  bases  for  describing  the  activities  to  be 
accomplished  in  the  identification  of  the  information  required  and  available  for  the 
vulnerability  assessment  of  a  weapon  system,  as  well  as  the  information  required  but 


*  A  system  is  composed  of  subsystems,  and  each  subsystem  is  a  set  of  components.  Critical  components 
are  those  components  that  if  lost  will  result  in  a  degradation  of  one  or  more  subsystem  functions  and, 
consequently,  a  reduction  in  system  capability  [25]. 

f  Figure  2  is  an  adaptation  of  the  Mission-Based  Acquisition  Strategy,  chart  by  Deitz  [26]. 
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9.  Compare  system  physical  status  in  activity  #5  to  status  in  #8 
Figure  2.  Taxonomy  of  Vulnerability  Assessment. 


unavailable  or  unreliable  (i.e.,  data  voids).  The  activities  listed  in  Figure  2  begin  at 

Level  4  and  are  described  as  follows: 

1.  Define  system  utility  status  (Level  4):  Define  the  likely  combat  scenarios  and  the 
role  played  by  the  weapon  system  in  successful  completion  of  identified 
mission(s).  Define  the  measures  of  battlefield  effectiveness  available  to  describe 
the  system  utility  or  mission  status  (i.e.,  mission  complete  or  not  complete).  The 
characteristics  of  likely  combat  scenarios  (i.e.,  physical  environment,  threat 
environment,  and  mission  objectives),  as  well  as  users'  tactics  and  doctrine 
relevant  to  the  system,  serve  as  input  to  this  activity.  In  Figure  2,  the  subspace 
shown  at  Level  4  includes  those  points  representing  outcomes  in  Space  4  that  are 
associated  with  mission  completion/ success. 

2.  Develop  mission  operators  (03  4):  Develop  the  operators  that  map  the  mission¬ 
relevant  system  capability  measures  in  Level  3  to  the  MOEs  in  Level  4.  The 
mapping  operators  establish  the  link(s)  between  the  m-tuple  capability  state 
vectors  representing  the  specific  m  number  of  capability  measures  to  be 
evaluated,  and  the  MOEs,  representing  the  mission  status  measures  of  battlefield 
success  (i.e., mission  complete  or  not  complete).  A  single  capability  state  vector 
in  Level  3  may  map  to  one  or  more  points  or  to  no  points  in  Level  4,  depending 
on  the  tasks  required  in  the  combat  scenarios  represented  in  Level  4.  In  other 
words,  a  set  of  aggregate  capabilities  may  lead  to  mission  success  in  one  or  more 
given  scenarios,  but  that  same  set  of  capabilities  may  not  lead  to  mission 
completion  in  other  scenarios  that  require  a  different  combination  of  capabilities. 

3.  Define  (mission-relevant)  system  capability  status  (Level  3):  Define  the  minimum 
level  of  each  of  the  m  specific  mission-relevant  system  capability  measures  (i.e., 
measures  identified  in  activity  #2)  that  are  required  for  mission  completion.  This 
activity  identifies  the  specific  m-tuple  capability  state  vectors  in  Level  3  (shown  as 
subspace  of  Level  3  in  Figure  2)  that  map  to  mission  completion  in  Level  4  for  a 
specific  combat  scenario.  Each  of  the  m  elements  in  the  capability  state  vector 
reflects  the  minimum  level  of  the  m  number  of  measures  of  specific  capabilities 
that  must  be  achieved  for  mission  completion. 
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4.  Develop  capability  operator(s)  (02f)-  Develop  the  operators  that  map  the  damage 
states  in  Level  2  to  the  measures  of  capability  in  Level  3.  The  mapping  operators 
or  degraded  state  operators  establish  the  link(s)  between  the  damage  state  vectors, 
representing  the  damage  states  of  the  system  following  threat-system  interaction, 
and  the  capability  state  vectors,  representing  the  capabilities  required  to 
complete  the  mission(s).  This  mapping  defines  the  components  and  subsystems 
that  must  be  in  operation  to  deliver  the  levels  of  capability  identified  in 
activity  #3. 

5.  Define  system  physical  status  (Level  2):  Define  the  specific  critical  components  that 
must  remain  in  operation  (i.e.,  not  killed)  after  threat-system  interaction  to  result 
in  the  identified  capability  levels  required  for  mission  completion.  Redundancies 
among  components  must  be  considered.  This  activity  identifies  the  specific  n- 
tuple  damage  state  vectors  of  Level  2  (shown  in  subspace  at  Level  2  in  Figure  2) 
that  map  to  the  required  m-tuple  capability  state  vectors  of  Level  3  (as  identified 
in  #3),  which  in  turn  map  to  mission  completion. 

6.  Define  initial  conditions  (Level  1):  Define  the  set  of  initial  configurations  of  the 
threat  (likely  to  be  encountered  in  combat)  and  weapon  system  just  prior  to 
interaction.  This  set  of  configurations  selected  considers  the  mission  objectives, 
the  physical  environment  of  combat,  and  the  users'  tactics  and  doctrine. 

7.  Develop  damage  operators  (01/2):  Develop  the  damage  operators  expected  in 
interactions  of  the  system  and  threat  (e.g.,  damage  mechanisms,  such  as  threat 
penetration,  fragment  penetration,  fire,  or  shock).  The  damage  operators 
establish  the  link(s)  between  the  initial  conditions  (Level  1)  of  the  threat  and 
weapon  system  prior  to  interaction  and  the  damage  state  vectors  of  Level  2, 
representing  the  system's  physical  status  following  threat-system  interaction. 


*  This  mapping  will  also  identify  the  specific  n-tuple  damage  state  vectors  in  Level  2  that  will  not  map  to 
the  identified  required  m-tuple  capability  state  vectors  in  Level  3. 
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8.  Define  system  physical  status  (Level  2):  Define  the  specific  n-tuple  damage  state 
vectors  (Level  2)  that  are  mapped  from  the  specific  points  (i.e.,  initial  conditions) 
of  threat-system  interaction  defined  in  Level  1.  The  n  elements  of  the  damage 
state  vector  describe  the  status  (i.e.,  kill,  no  kill)  of  the  n  number  of  components 
of  the  system  following  a  threat-system  interaction. 

Analysts  need  to  be  able  to  predict  not  only  the  interactions  that  are  likely  to 
result  in  the  loss  of  critical  components,  but  also  the  interactions  that  are  likely  to 
result  in  personnel  casualties  (i.e.,  personnel  vulnerability).  Personnel  casualties 
may  also  affect  the  probability  of  mission  completion  (i.e.,  specific  number  of 
fully  functioning  crew  members  may  represent  a  critical  component  of  system). 
This  activity  identifies  system  vulnerabilities  associated  with  personnel 
casualties,  as  well  as  the  potential  for  catastrophic  loss  of  the  system. 

9.  Compare  system  physical  status  in  activity  #5  to  status  in  #8:  Compare  the 
physical  status  of  the  system  required  for  mission  completion  (activity  #5)  and 
the  identified  physical  status  of  the  system  following  threat-system  interaction 
(activity  #8).  This  activity  compares  (1)  the  specific  critical  components  that 
must  remain  in  operation  (i.e.,  not  be  killed)  after  the  threat-system  interaction  to 
deliver  the  identified  capability  levels  required  for  mission  completion  to  (2)  the 
specific  critical  components  that  are  expected  to  remain  in  operation  following 
the  threat-system  interaction. 

This  activity  identifies  system  vulnerabilities  associated  with  a  system's  failure  to 
complete  its  mission. 

4.2.3  Data  Sources.  The  Taxonomy  VA  clearly  illustrates  that  the  data  required 
for  the  assessment  of  system  vulnerability  fall  in  one  of  three  categories:  (1)  Data  Set  A 
(see  Figure  2),  data  that  link  the  mission-relevant  capability  measures  of  the  system  to 
the  mission-status  measures  of  battlefield  success  (03/4)  in  combat  scenarios  likely  to  be 
encountered  by  the  system,  given  users'  tactics  and  doctrine  associated  with  those 
scenarios;  (2)  Data  Set  B  (see  Figure  2),  data  that  link  the  damage  state  vectors  resulting 
from  threat-system  interaction  to  the  capability  state  vectors  associated  with  mission 
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success  (Ow);  or  (3)  Data  Set  C  (see  Figure  2),  data  that  link  the  initial  conditions  of  the 
threat-system  interaction  to  the  resulting  damage  state  vectors  (01/2).  Sources  for 
obtaining  the  required  data  are  explored  in  this  section. 

To  begin  the  vulnerability  assessment  process,  the  analyst  must  have  a  thorough 
understanding  of  the  combat  scenarios  in  which  the  system  is  likely  to  operate  (i.e.,  the 
physical  environment,  the  likely  threats,  and  the  mission[s]  to  be  completed)  and  the 
relevant  users'  tactics  and  doctrine.  A  thorough  analysis  of  likely  scenarios  forms  the 
basis  for  determining  the  minimum  levels  of  capabilities  (i.e..  Level  3  output)  needed  in 
the  set  of  combat  scenarios  selected  (i.e.,  activities  #1-3  in  preceding  analysis).  Sources 
for  this  information  include  the  designated  users  of  the  weapon  system,  military 
strategists,  and  analysts  and  operations  researchers  associated  with  the  evaluation 
agencies  of  the  military  service. 

It  is  proposed  that  a  major  objective  of  vulnerability  assessment  is  to  evaluate  the 
extent  to  which  a  weapon  system  retains  the  capabilities  that  were  determined  at  the  time 
of  acquisition  to  be  needed  for  mission  success  when  the  system  interacts  with  threats  it  is 
likely  to  encounter  in  combat.  It  is  assumed,  therefore,  that  the  data  defining  the 
relationship  between  the  mission-relevant  capability  measures  and  the  mission-status 
measures  of  battlefield  success  serve  as  input  to  decisions  associated  with  the  design 
and  engineering  of  the  system.  The  operational  requirements  for  the  system, 
incorporating  the  levels  of  relevant  capabilities  needed  by  the  system  for  mission 
success,  define  the  minimum  levels  of  those  capabilities  that  must  be  retained  by  the 
system  in  combat  and  provide  the  basis  for  determining  the  data  required  for  the 
vulnerability  assessment.  These  data  (i.e..  Data  Set  A  in  Figure  2)  should  be  readily 
available  to  personnel  designing  the  vulnerability  assessment  plan. 

To  complete  the  assessment,  the  analyst  must  obtain  data  relevant  to  (1)  the 
relationship  between  the  damage  vectors  and  the  measures  of  capability  required  by  the 
system  to  complete  its  mission  (02f)  and  (2)  the  relationship  between  the  threat-system 
interaction  and  the  damage  vectors  associated  with  system  and  personnel  vulnerability 
(Ou).  Potential  data  sources  include  (1)  results  of  prior  tests  of  materials,  components, 
subsystems  of  earlier  and  current  models  of  the  system  or  similar  systems,  (2)  results  of 
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prior  system-level  tests  of  earlier  models  of  the  system  or  similar  systems,  (3)  combat 
data  relevant  to  damage  mechanisms,  system  damage,  and  residual  capabilities  of 
system  as  associated  with  the  identified  threats,  (4)  design  analyses  of  the  system  with 
consideration  given  to  the  new  materials  and  technologies  incorporated  into  the  system, 
(5)  prior  engineering  analyses  or  CDTs,  and  (  6)  M&S  runs  that  incorporate  the  system 
description,  threat  characteristics,  and  damage  mechanisms  expected  in  threat-system 
interactions.  These  data  (i.e..  Data  Sets  B  and  C  in  Figure  2)  would  be  sought  from 
defense  databases,  the  system  contractor,  and  defense  analysis,  testing,  and  evaluation 
agencies.  Information  relevant  to  the  01/2  or  the  02/3  mappings  that  is  either  unavailable 
from  the  identified  sources  or  is  not  considered  reliable  by  analysts  defines  the  subset  of 
the  required  information  (i.e.,  data  voids)  to  be  addressed  in  the  vulnerability 
assessment  plan.* 

4.2.4  Critical  Data  Voids.  Many  resources  would  be  required  to  design  and 
conduct  a  vulnerability  assessment  plan  that  explores  all  data  voids  associated  with  all 
possible  combat  scenarios.  To  ensure  an  efficient  and  effective  use  of  limited  resources, 
analysts  must  prioritize  the  data  voids  to  be  addressed  in  the  assessment  plan  (i.e., 
establish  critical  data  voids).  The  first  cut  in  restricting  data  voids  to  be  addressed  is 
achieved  by  limiting  the  preceding  analysis  of  activities  to  the  combat  scenarios  that  are 
more  likely  to  unfold.  Analysts  must  also  consider  the  degree  of  confidence  placed  by 
decision-makers  in  the  subset  of  required  information  that  is  available.  Figure  3 
provides  one  representation  of  an  approach  in  which  priorities  (one  [1]  representing 
high  priority  and  four  [4]  representing  low  priority)  are  assigned  to  several  groups  of 
data  voids  to  be  addressed  in  a  vulnerability  assessment  plan. 

A  data  void  in  the  information  set  required  for  vulnerability  analysis  is  assumed  to 
exist  if  (1)  data  are  not  available  or  (2)  the  confidence  in  available  data  is  not  high 
(i.e.,  low  to  moderate  confidence  only  in  available  data).  In  designing  an  effective 


*  It  is  assumed  that  the  vulnerability  assessment  program  is  begun  early  in  the  acquisition  cycle.  Some  of 
the  identified  information  sources  may  have  relevant  data  in  place  at  the  beginning  of  the  planning 
process;  other  sources  may  produce  data  as  part  of  a  vulnerability  assessment  plan  executed  to  obtain 
data  relevant  to  evaluators'  decisions. 
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vulnerability  assessment  plan  with  limited  resources,  higher  priorities  would  be 
assigned  to  addressing  Ow  or  023  data  voids  associated  with  reasonably  possible  or 
probable  combat  scenarios  (i.e.,  likely  combat  scenarios)  in  which  confidence  in 
available  data  is  not  high  (i.e.,  cells  labeled  [1]  and  [2]).  Gathering  data  relevant  to  Ou 
and  02/3  mappings  in  likely  scenarios  in  which  there  is  high  confidence  in  available  data 
(i.e.,  cells  labeled  [3])  may  be  part  of  an  assessment  plan  if  funds  are  obtainable  for 
assurance  testing.  Generally,  a  lower  priority  would  be  assigned  to  the  analysis  of 
activities  and  data  voids  associated  with  combat  scenarios  considered  only  remotely 
possible  (i.e.,  cells  labeled  [4]).‘  Although  classifying  data  voids  into  four  categories 
may  facilitate  a  systematic  approach  to  addressing  data  voids,  priorities  may  well  need 
to  be  defined  with  finer  resolution  as  resources  become  more  limited. 

In  this  report,  it  is  assumed  the  data  voids  to  be  addressed  in  an  environment  of 
limited  resources  are  restricted  to  that  subset  of  data  voids  associated  with  likely 
combat  scenarios  in  which  information  relative  to  012  and  023  mappings  is  either 
unreliable  or  unavailable  (i.e.,  priorities  [1]  and  [2]  in  Figure  3).  Prioritization  of  specific 
data  voids  within  this  restricted  subset  may  be  accomplished  by  either  engaging  in 
subjective  forms  of  analyses  or  employing  quantitative-qualitative  analytical  tools,  such 
as  the  Analytic  Hierarchy  Process  (AHP)  or  Quality  Function  Deployment  (QFD).  A 
description  and  discussion  of  the  applicability  of  these  tools  to  group  decision-making 
may  be  found  in  other  sources  [27-32]. 

4.3.  Vulnerability  Assessment  Plans.  A  cost-effective  strategy  to  obtain  data 
needed  to  assess  the  vulnerability  of  a  weapon  system  prior  to  fielding  requires  (1)  a 
clear  identification  of  the  data  required  for  the  decisions  to  be  made,  the  data  available 
from  reliable  sources,  and  the  data  remaining  to  be  gathered  (i.e.,  data  voids)  and  (2)  a 
vulnerability  assessment  plan  designed  to  address  the  data  voids  with  consideration 
given  to  the  limited  resources  available.  The  methodology  proposed  in  this  report  for 
the  selection  of  the  optimal  vulnerability  assessment  plan  from  a  group  of  alternative 


'  It  is  acknowledged  that  analysts  may  choose  to  assess  the  vulnerabilities  of  combat  scenarios  that  are 
only  remotely  possible  if  the  scenarios  are  associated  with  very  costly  losses  and  the  resources  for 
assessment  are  available. 
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assessment  plans  incorporates  many  of  the  principles  of  a  Cost  as  an  Independent 
Variable  (CAIV)  strategy. 


4.3.1  CATV  Principles.  The  Defense  Acquisition  Deskbook  describes  CAIV  as 
follows  [33]: ' 

CAIV  is  a  strategy  that  entails  setting  aggressive,  yet  realistic  cost 
objectives  when  defining  operational  requirements  and  acquiring  defense 
systems  and  managing  achievement  of  these  objectives.  Cost  objectives 
must  balance  mission  needs  with  projected  out-year  resources,  taking  into 
account  existing  technology,  maturation  of  new  technologies  and 
anticipated  process  improvements  in  both  DoD  and  industry. 

The  CATV  methodology  supports  the  setting  of  realistic  but  aggressive  cost 
objectives  early  in  the  acquisition  program  and  the  implementation  of  cost-reduction 
strategies  throughout  the  program,  including  cost-performance  tradeoffs,  competition 
among  suppliers /contractors,  integration  of  program  activities  through  early  planning, 
and  the  development  of  efficiencies  in  core  activities  [34]. 


Characteristics  of  the  CAIV  methodology  follow  [34, 35]: 

•  Cooperation  of  PMO,  system  user,  contractor/manufacturer  of  system,  Service 
leaders,  and  OSD  is  required  for  successful  implementation  in  the  production 
decision.  System  users  play  an  active  role  in  the  tradeoff  process  throughout  the 
life  cycle  of  the  system. 

•  Analyses  of  measures  of  mission  effectiveness  vs.  performance  requirements  and 
performance  requirements  vs.  costs  of  systems  are  conducted  to  support 
production  decisions. 

•  The  program  budget  defines  available  funds  and  drives  the  acquisition  strategy 
for  the  program.  Cost  goals  must  meet  objectives  of  affordability  and  ability  to 
fund  threshold  requirements  of  system. 

*  Implementation  of  CAIV,  an  initiative  to  reduce  life-cycle  costs  of  defense  systems,  is  required  of  all 
new  AC  AT  I  and  IA  programs  [33]. 
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•  System  performance  parameters  (included  in  ORD)  are  the  designated  key 
performance  parameters.  Potential  for  tradeoffs  of  cost  against  performance  is 
increased  if  the  number  of  key  performance  parameters  is  minimized.  Key 
performance  parameters  are  not  tradable  below  a  threshold  value,  and 
parameters  must  meet  users'  needs. 

•  Tradeoff  process  begins  early  in  the  design  process  with  updates  to  costs  and 
performance  requirements  incorporated  throughout  implementation.  Metrics  to 
track  performance  are  established.  Long-range  planning  is  implemented. 

•  Risks  (uncertainties)  in  CATV  implementation  over  program  life  cycle  include 
the  stability  of  the  program  budget  and  the  priorities  for  the  system,  (2)  the 
stability  of  the  mapping  of  required  measures  of  effectiveness  to  minimum 
performance  requirements,  (3)  the  availability  of  databases  and  models 
applicable  to  cost  estimations  for  systems,  (4)  the  potential  for  tradeoffs  of 
performance  requirements,  costs,  and  schedules,  (5)  the  analysts'  capacity  to 
understand  the  interrelationships  of  performance  requirements,  (6)  the 
incorporation  of  performance  requirements  in  production  contract  with 
appropriate  incentives,  and  (7)  the  developments  in  technology  to  enable 
achievement  of  program  objectives. 

The  core  of  the  CAIV  methodology  is  the  tradeoff  process,  in  which  mission  needs 
are  balanced  against  projected  resources  with  consideration  given  to  scheduling  issues 
and  identified  risks.  In  production  decisions  employing  CATV  principles,  risks  are 
managed  to  achieve  cost,  schedule,  and  performance  objectives. 

In  this  report,  it  is  proposed  that  the  selection  of  the  optimal  vulnerability 
assessment  plan  be  made  by  trading  off  the  plan's  performance  (i.e.,  the  plan's  capacity 
to  assess  the  critical  issues  of  personnel/ system  vulnerability)  against  the  budget  for 
plan  implementation  with  consideration  given  to  scheduling  issues  and  risks  identified 
in  achieving  the  plan.  Key  parameters  are  identified  as  the  critical  prioritized  data 
voids  to  be  addressed  in  the  assessment  plan.  Major  analyses  would  include 
performance  and  cost  estimates  for  the  vulnerability  assessment  plans  and  the  elements 
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of  the  assessment  plans.  The  methodology  proposed  employs  many  of  the  CATV 
principles  described  in  the  context  of  production  decisions  in  the  prior  paragraphs  of 
this  section  and  emphasizes  the  exploration  of  alternative  assessment  strategies  to 
achieve  vulnerability  assessment  objectives. 

4.3.2  Construction  of  Vulnerability  Assessment  Plan.  The  Taxonomy  VA  (Figure  2) 
presents  a  framework  for  identifying  the  data  voids,  a  subset  of  the  information 
required  to  conduct  a  vulnerability  assessment  of  a  weapon  system.  To  complete  the 
vulnerability  assessment,  analysts  must  design  a  blueprint,  a  vulnerability  assessment 
plan,  to  address  those  data  voids.  The  design  of  an  assessment  plan  requires  the 
identification  of  potential  sources  or  activities  (i.e.,  elements  of  the  plan)  for  obtaining 
data  to  address  critical  data  voids  and  the  evaluation  of  the  capacity  of  each  element  to 
produce  the  critical  data  within  the  time  constraints  of  the  assessment  process. 
Assuming  alternative  vulnerability  assessment  plans  to  provide  the  critical  data  are 
available  and  the  budgets  for  obtaining  the  data  are  limited,  the  analyst  must  select  the 
optimal  (i.e.,  cost-effective)  vulnerability  assessment  plan  from  the  group  of  alternative 
plans  by  weighing  (1)  the  capability  or  capacity  of  each  plan  to  provide  the  critical  data 
(i.e.,  benefits  of  plan)  and  (2)  the  cost  to  execute  all  elements  of  the  plan  with 
consideration  given  to  the  uncertainty  (risk)  associated  with  each  plan's  capacity  to 
provide  the  critical  data  at  the  identified  cost  within  the  time  constraints  of  the 
assessment  process.* 

Constructing  a  vulnerability  assessment  plan  requires  the  identification  of  potential 
sources  or  activities  that  could  provide  data  to  address  the  critical  data  voids.  These 
activities  serve  as  elements  of  the  assessment  plan  and  may  include  modeling  and 
simulation;  controlled  damage  testing;  experimental  testing;  developmental  testing; 
design  and  engineering  analyses;  component- ,  subsystem- ,  and  system-level  live-fire 
testing;  and  FU  SL  LFT.  Factors  that  provide  input  to  the  design  of  the  vulnerability 
assessment  plan  include  the  nature  of  the  identified  prioritized  critical  data  voids,  the 


*  Uncertainty  (risk)  associated  with  a  plan's  capacity  to  provide  the  critical  data  within  the  time 
constraints  at  tire  identified  cost  includes  the  uncertainty  associated  with  the  availability  of  system 
hardware  needed  for  execution  of  the  plan. 
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budget  for  assessment,  the  availability  of  hardware,  the  timetable  for  the  completion  of 
the  assessment  process,  and  other  constraints  unique  to  the  identified  data  sources. 

For  example.  Figure  4  shows  alternative  elements  that  may  have  the  potential  to 
address  identified  data  voids.  Each  element  would  be  described  in  detail  and  the  cost 
of  executing  the  element  estimated.  Figure  5  illustrates  three  vulnerability  assessment 
plans,  each  including  several  alternative  elements  to  address  the  identified  critical  data 
voids.  Each  plan  includes  a  different  combination  of  elements  and  is  associated  with  a 
different  total  cost.* 

4.3.3  Capability  of  Vulnerability  Assessment  Plan.  In  designing  a  vulnerability 
assessment  plan  to  address  critical  data  voids,  an  important  question  arises:  "How  does 
one  measure  the  capability  or  the  performance  of  a  plan  that  is  designed  to  investigate 
the  unknown?"  This  is  a  difficult  question  to  answer  ex  ante  or  ex  post  execution  of  the 
plan.  An  objective  measure  of  capability  is  not  available  for  this  type  of  analysis,  and  an 
appropriate  subjective  measure  is  sought  to  define  the  capability  or  performance  of  the 
plan.  As  previously  noted,  in  selecting  the  optimal  vulnerability  assessment  plan,  the 
analyst  must  consider  not  only  the  capacity  of  a  plan  to  produce  the  critical  data,  but 
also  the  uncertainty  associated  with  the  plan's  ability  to  supply  the  required  data  within 
the  time  and  financial  constraints  of  the  assessment  environment. 

A  subjective  metric  of  risk  assessment  is  proposed  for  evaluating  the  capability  of  a 
vulnerability  assessment  plan.  Thus,  the  capability  of  a  plan  is  rated  by  the  degree  of 
uncertainty  a  designated  panel  of  experts  associates  with  the  plan's  capacity  to 
adequately  address  the  prioritized  critical  data  voids  within  the  available  time  for 
assessment  at  the  identified  cost.  A  four-point  scale  of  risk  assessment  is  proposed  with 
ratings  of  high,  medium-high,  medium,  and  low  risk.  An  example  is  provided  in  the 
following  paragraphs. 


‘  If  the  elements  of  an  assessment  plan  (e.g.,  modeling,  experimental  testing,  controlled  damage  testing) 
differ  in  detail  (i.e.,  different  models  are  exercised  within  modeling  component,  different  tests  are 
included  in  experimental  testing),  the  vulnerability  assessment  plans  also  differ. 
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Figure  4.  Potential  Elements  of  Vulnerability  Assessment  Plan  to 
Address  Critical  Data  Voids  (DVs). 
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Assume  an  expert  panel  has  assessed  the  risk  associated  with  the  three  vulnerability 
assessment  plans  illustrated  in  Figure  5.  The  described  four-point  scale  is  employed  to 
assess  the  risks  associated  with  each  plan's  capability  to  address  the  specific  identified 
data  voids.  As  illustrated  in  Figure  6,  the  expert  panel  assesses  a  low  risk  (i.e.,  little 
uncertainty)  to  be  associated  with  the  capacity  of  Vulnerability  Assessment  Plan  #1  to 
adequately  address  Data  Voids  #3  and  #5,  but  a  medium  risk  to  be  associated  with 
addressing  Data  Voids  #1,  #2,  and  #4.  The  low  risk  assessment  assigned  to  the 
Vulnerability  Assessment  Plan#l  and  Data  Void  #3  intersecting  cell  is  interpreted  as 
little  uncertainty  associated  with  the  capacity  of  Vulnerability  Assessment  Plan  #1  to 
adequately  address  Data  Void  #3  within  the  given  time  frame  at  the  specified  cost. 

4.3.4  Cost  of  Vulnerability  Assessment  Plan.  An  important  consideration  in  the 
selection  of  the  optimal  assessment  plan  is  the  total  estimated  cost  of  accomplishing  the 
identified  elements  of  the  alternative  plans.  As  discussed  in  section  3  of  this  report,  a 
valid  comparison  of  alternative  assessment  strategies  requires  the  establishment  of 
accounting  procedures  to  ensure  consistency  and  comparability  in  estimating  costs 
across  plans,  as  well  as  across  periods  and  systems.  Decision-makers  must  be  provided 
with  cost  data  relevant  to  all  elements  of  each  alternative  plan  prepared  according  to  a 
cost  methodology  applied  consistently  across  all  plans.  This  methodology  includes 

•  Identification  of  all  tasks  included  in  the  elements  of  each  plan  and  the  cost 
components  of  those  tasks  (e.g.,  direct  materials  and  labor,  indirect 
materials  and  labor,  facilities  costs,  etc.). 

•  Identification  of  the  behavior  of  identified  costs  (e.g.,  cost  varies  with  volume, 
cost  fixed  over  relevant  range  of  volume,  etc.). 

•  Identification  and  description  of  method(s)  employed  to  measure  costs,  if 
alternative  methods  are  available;  disclosure  of  alternative  values,  if  available. 

•  Identification  of  all  allocated  costs  and  bases  for  cost  allocation. 

•  Identification  and  explanation  of  all  incomplete  cost  data. 
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There  are  benefits  in  identifying  the  costs  of  the  individual  elements  of  an 
assessment  plan  and  the  costs  of  the  tasks  that  are  included  in  each  element.  Activity- 
Based  Costing  (ABC),  a  costing  methodology  that  focuses  on  the  activity  or  task  as  the 
basic  cost  objective,  has  been  widely  used  in  recent  years  to  determine  a  more  accurate 
cost  of  producing  a  product  or  delivering  a  service.  Employing  ABC  to  cost  the 
elements  of  an  assessment  plan  requires  the  identification  of  all  tasks  in  each  plan 
element  that  consume  resources  and  the  total  costs  associated  with  each  task.  Tasks 
must  be  carefully  examined  to  identify  the  tasks'  cost  drivers,  the  factors  that  most 
closely  drive  or  cause  the  costs  to  increase  or  decrease. 

The  advantage  of  employing  ABC  to  cost  individual  elements  of  an  assessment  plan 
is  that  ABC  provides  a  means  to  weigh  the  value  added  against  the  costs  incurred  for 
each  plan  element.  ABC  affords  the  framework  to  identify  tasks  of  an  element  that  are 
non-value-added  and  are  able  to  be  eliminated  as  well  as  tasks  that  are  value-added  but 
are  able  to  be  made  more  efficient  [36-38]. 

4.3.5  Selection  of  Optimal  Vulnerability  Assessment  Plan.  To  select  the  optimal 
(i.e.,  cost-effective)  vulnerability  assessment  plan  from  a  group  of  alternative  plans,  the 
decision-makers  must  define  (1)  the  maximum  levels  of  risk  in  addressing  each  of  the 
critical  data  voids  they  are  willing  to  accept  in  a  vulnerability  assessment  plan  and 
(2)  the  maximum  cost  they  are  willing  to  incur  in  plan  execution. 

The  priorities  assigned  to  the  identified  data  voids  serve  as  input  to  decisions 
relevant  to  the  risk  assessment  levels  evaluators  require  in  an  acceptable  assessment 
plan.  For  example,  assume  analysts  (using  the  four-point  scale  of  risk  assessment 
previously  described)  have  determined  certain  maximum  levels  of  risk  they  are  willing 
to  accept  in  a  vulnerability  assessment  plan  (i.e.,  plan  addressing  five  prioritized  data 
voids).  Those  maximum  levels  of  acceptable  risk  are  shown  in  Figure  7.  Higher 
priorities  assigned  to  critical  data  voids  would  be  associated  with  lower  maximum 
levels  of  risk  acceptable  to  the  analysts.  In  the  example  described  in  Figure  7,  a  plan  is 
termed  acceptable  (i.e.,  with  respect  to  performance),  if  analysts  associate  a  low  risk 
with  the  plan's  capacity  to  adequately  address  Data  Voids  #1  and  #2  and  medium 


66 


II 

II 

II 

II 

CD 

”0 
•  fH 

o 

> 

IS 

D 

"3 

u 


9h 

u 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

v 


.CO 

ID 

=#= 

> 

£ 

s 

0 

*  T^A 

’S 

£ 

.CD 

% 

s 

> 

S 

0 

'  T-^ 

l 

40 

CO 

=£ 

> 

£ 

a 

0 

*  T**i 

2 

a; 

cs 

.cc 

% 

> 

$ 

Q 

o 

Nj 

rH 

CD 

* 

£ 

> 

5 

0 

o 

Kj 

§  *2* 
a  *S  ^ 

g»  a  w 
•1  a.  ^ 

«  y  w 

•CN  U  ■ ’•* 
<  « 

CD 

CD 

2 


£ 


£ 

0> 

6 

CD 

CD 

0> 

CD 

CD 

c 


CD 

bo 

£ 

rt 

& 

X 

« 

O 

hJ 

*d 

s 

s' 

s 


•w* 

u 

a> 

£ 

3 

> 

c 


W> 


6 

s 

a? 


a> 

l-H  Jh 

•o  bo 

8| 

u  ” 

<  O 
.*  > 
.2  «« 
esJ  « 

mh  Q 

°  Zl 

CD  ffS 

*3  .a 

a!  T3 
hJ  U 
g  *T3 
§.2 
S5 

X  c 

fS  ^ 


a 

£ 

bO 


67 


(or  less)  risk  with  the  plan's  capacity  to  adequately  address  Data  Voids  #3,  #4,  and  #5 
within  the  time  constraints  of  the  vulnerability  assessment  process  and  at  the  specified 

cost  of  plan  execution. 

The  measure  of  a  successful  vulnerability  assessment  depends  on  which  data  voids 
are  addressed  by  the  vulnerability  assessment  plan,  the  capacity  of  the  assessment  plan 
to  address  those  data  voids  as  measured  by  the  risk  assessment  metric,  and  the 
priorities  attached  to  those  data  voids  by  the  evaluators  of  the  vulnerability  of  the 
system.  The  capabilities  required  of  an  acceptable  vulnerability  assessment  plan  as 
measured  by  the  maximum  risk  levels  allowed  must  be  compared  to  the  capability 
measures  of  the  alternative  vulnerability  assessment  plans  proposed  for  consideration. 
Figure  8  provides  a  comparison  of  the  risk  assessment  ratings  of  Vulnerability 
Assessment  Plans  #1,  #2,  and  #3  (see  Figure  6)  to  the  maximum  levels  of  risk 
determined  to  be  acceptable  (see  Figure  7).  The  comparison  reveals  both  Vulnerability 
Assessment  Plans  #2  and  #3  are  acceptable  in  terms  of  expected  plan  performance .  A 
comparison  of  the  expected  costs  of  completing  either  of  the  two  plans  to  the  maximum 
amount  allowed  in  the  budget  for  plan  execution  also  finds  both  plans  to  be  acceptable. 

The  comparison  of  alternative  assessment  plans  affords  decision-makers  an 
opportunity  to  look  at  the  tradeoffs  in  alternative  plans  relative  to  cost  and  performance 
(i.e.,  capacity  of  the  plan  to  adequately  assess  vulnerabilities  associated  with  the  data 
voids).  Although  Vulnerability  Assessment  Plan  #3  initially  appears  to  be  the  optimal 
plan  (i.e..  Vulnerability  Assessment  Plan  #3  meets  maximum  levels  of  risk  allowed  of 
plan  and  has  a  lower  cost  than  Vulnerability  Assessment  Plan  #2),  possible  questions 
for  the  decision-makers  may  be 

•  Realizing  that  one  remains  within  the  budgetary  maximum  of  $20,  is  one  willing 
to  pay  $2  more  and  adopt  VAP  #2,  thereby  decreasing  the  level  of  risk  in 
addressing  Data  Void  #3  from  moderate  to  low? 

•  Is  one  willing  to  adopt  VAP  #1  and  accept  more  risk  in  addressing  Data 
Voids  #1  and  #2  (and  less  risk  in  addressing  Data  Voids  #3  and  #5)  to  save  $3? 
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•  If  the  costs  of  VAP  #1  are  allowed  to  increase,  are  the  risk  levels  of  VAP  #1 
lowered  to  acceptable  levels? 

It  is  important  to  realize  that  the  estimated  costs  of  all  alternative  vulnerability 
assessment  plans  may  be  over  and  above  the  amounts  allocated  or  budgeted  for  the 
vulnerability  assessment  program.  It  is  at  this  point  that  program  managers  must 
carefully  weigh  the  pros  and  cons  of  reallocating  dollars  from  other  programs  to  V/L 
assessment  or  of  requesting  additional  funds  for  conducting  an  assessment  plan  that 
meets  the  performance  standards  (i.e.,  allowable  levels  of  risk  in  addressing  critical  data 
voids)  determined  to  be  acceptable  to  decision-makers. 


4.4  Complexities  of  Methodology.  There  are  several  complexities  inherent  in  the 
proposed  approach  to  vulnerability  assessment  that  must  be  addressed.  First,  the 
capability  or  the  capacity  of  the  vulnerability  assessment  plan  to  adequately  address 
critical  data  voids  is  rated  by  a  subjective  measure,  the  assessment  of  risk  in  the  plan's 
capacity.  The  lack  of  an  objective  measure  to  rate  the  plan's  ability  to  provide  all  of  the 
data  relative  to  identified  data  voids  or  address  the  unknown  unknowns  is  not  a 
problem  unique  to  weapon  system  analysis/  In  many  disciplines  (e.g.,  finance,  law, 
and  medicine),  opinions  of  expert  panels  or  groups  of  individuals  with  experience  in 
similar  decision-making  tasks  are  sought  for  evaluations  in  which  objective  measures 
are  not  available/'  Group  assessment  or  evaluation  (i.e.,  panel  of  experts)  is  proposed 


A  complete  analysis  that  includes  ALL  possible  combinations  of  data  source  components  that  have 
associated  costs  within  the  budget  range  would  include  the  suggested  alternative  vulnerability 
assessment  plan  (i.e.,  increased  costs  and  decreased  risk  levels).  However,  given  the  limitations  in 
information  processing  of  individuals,  this  type  of  comparison  affords  a  check  on  the  completeness  of 
alternatives  generated  by  analysts/testers/evaluators. 

t  Xo  employ  an  objective  measure  to  assess  the  capability  of  a  vulnerability  assessment  plan  to  provide  all 
of  the  data  relative  to  critical  data  voids  would  require  knowledge  or  definition  of  all  of  the  data. 
Knowledge  of  all  unknowns  would  be  necessary  to  answer  the  question,  "Were  all  the  unknown 
unknowns  identified?"  Although  components  of  vulnerability  assessments  plans  may  be  compared 
(e.g.,  vulnerabilities  detected  in  component  testing  versus  vulnerabilities  detected  in  modeling),  a 
comparison  of  even  several  vulnerability  assessment  plans  would  be  economically  unfeasible. 

Therefore,  the  decision-maker  must  search  for  the  next  best  solution,  the  opinions  of  experts  as 
surrogate  for  the  objective  measure  of  capability. 

*  Recent  studies  in  other  disciplines  have  shown  that  experts  with  domain-specific  experience  often  make 
decisions  more  compatible  with  those  of  statistical  models  than  do  experts  with  more  global  experience. 
To  avoid  unneeded  bias,  it  is  imperative  that  care  be  given  to  the  choice  of  experts  invited  to  join  the 
decision  process  [39, 40]. 
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because  numerous  studies  have  established  that  group  information-processing  often 
results  in  judgments  that  are  superior  to  the  judgments  of  individuals  [41]. 

A  second  complexity  arises  in  the  determination  and  comparison  of  the  costs  of 
vulnerability  assessment  plans.  At  the  present  time,  the  costs  of  testing  and  evaluation 
of  weapon  systems  are  not  reported  consistently  across  services,  weapon  systems,  or 
time  periods.  The  computation  and  recording  of  costs  of  components  of  vulnerability 
assessment  plans  according  to  a  single  methodology  that  is  consistent  across  periods 
and  comparable  across  weapon  systems  would  provide  a  useful  database  for  analysts 
designing  vulnerability  assessment  plans  in  the  future.  Suggestions  for  addressing  this 
issue  have  been  discussed  in  this  report. 

Finally,  it  is  recognized  that  implementation  of  the  proposed  methodology  requires 
the  commitment  of  time  and  personnel  to  the  design  and  evaluation  of  alternative 
assessment  strategies.  A  cost-effective  vulnerability  assessment  strategy  is  an  important 
part  of  a  cost-effective  acquisition  process.  Failure  to  commit  funds  to  the  prevention  and 
detection  of  vulnerabilities  early  in  the  design  and  production  phases  of  the  system  may 
well  result  in  the  expenditure  of  much  larger  dollar  amounts  in  system  design  and 
engineering  modifications  or  retrofits  following  production.  This  issue  is  discussed  in 
more  depth  in  the  Conclusion. 

5.  Conclusion 

5.1  Role  of  FU  SL  LFT&E  in  Cost-Effective  Vulnerability  Assessment.  A 

methodology  to  identify,  measure,  and  categorize  the  costs  and  benefits  of  FU  SL 
LFT&E,  as  well  as  other  elements  of  a  vulnerability  assessment  plan,  is  presented  in 
prior  sections  of  this  report.  As  an  element  of  an  assessment  plan,  however,  FU  SL 
LFT&E  is  unique.  For  many  systems,  FU  SL  LFT&E  is  a  congressionally  mandated 
activity  conducted  by  independent  agencies*  prior  to  the  system  entering  full-scale 
production  [1].  Evaluating  the  role  of  FU  SL  LFT&E  in  a  cost-effective  vulnerability 

*  Activities  conducted  by  independent  agencies  are  defined  as  activities  not  performed  under  the 
direction  of  the  manufacturer/ contractor(s)  of  the  system  or  the  PMO  associated  with  the  system. 
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assessment  strategy  requires  an  understanding  of  the  vulnerability  assessment  process 
and  the  risks  inherent  in  the  process.’ 

5.1.1  Cost-Effective  Vulnerability  Assessment  Process.  A  cost-effective  vulnerability 
process  emphasizes  the  achievement  of  assessment  objectives  within  the  constraints  of 
resources  available.  This  report  proposes  the  effectiveness  and  efficiency  of  an 
assessment  strategy  may  be  increased  by  the  identification  of  the  specific  set  of  data 
required  for  vulnerability  assessment,  the  determination  of  the  subset  of  the  required 
data  currently  available  from  reliable  sources,  and  the  selection  of  a  cost-effective 
vulnerability  assessment  plan  to  fill  the  data  voids.  Because  the  design  of  vulnerability 
assessment  plans  is  based  not  only  on  the  capacity  of  the  plan's  elements  to  address  the 
critical  data  voids  but  also  on  (1)  the  time  constraints  of  the  assessment  schedule,  (2)  the 
production  schedule  and  availability  of  system  hardware,  and  (3)  the  costs  attached  to 
the  execution  of  the  plan,  there  are  often  tradeoffs  among  plan  performance  (i.e., 
capacity),  cost,  and  schedule  that  must  be  explored  before  the  optimal  plan  is 
determined. 

A  cost-effective  strategy  of  vulnerability  assessment  begins  at  the  Concept 
Exploration  phase  and  continues  through  Program  Definition  and  Risk  Reduction  and 
Engineering  and  Manufacturing  Development  phases  of  the  acquisition  process.  It  may 
include  experimental  testing,  M&S,  design  and  engineering  analyses,  controlled 
damage  testing,  component  and  subsystem  LFT,  FU  SL  LFT,  as  well  as  other  activities 
that  address  the  identified  critical  data  voids  relevant  to  assessment  decisions. 

Planning  early  for  vulnerability  assessment  may  increase  the  number  of  options 
available  to  address  the  critical  data  voids  and  may  allow  assessment  activities  to  be 
completed  early  in  the  acquisition  process  at  a  time  when  design  or  engineering 
changes  are  more  feasible  and/ or  more  economical.  For  example,  technical  tests 
conducted  early  give  insight  into  design  and  engineering  issues  and  often  employ  test 
articles  (e.g.,  mock-ups,  replicas,  etc.)  that  are  less  expensive  than  the  realistic  test 


*  To  facilitate  discussion  of  the  methodology  presented  in  section  4  of  the  report,  the  Conclusion  is 
restricted  to  a  discussion  of  cost-effective  vulnerability  assessment  issues. 
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articles  required  to  be  used  in  an  FU  SL  LFT.  There  are,  however,  some  critical  data 
voids  (e.g.,  cascading  and  synergistic  damage  mechanisms)  that  are  unable  to  be 
addressed  by  alternative  activities  and  unable  to  be  explored  until  late  in  the  production 
process  when  the  required  hardware  for  testing  is  available. 

5.1.2  Risks  of  Vulnerability  Assessment  Process.  There  is  a  risk  in  the  vulnerability 
assessment  process  that  no  matter  what  actions  are  taken  during  the  acquisition  process 
by  the  contractor,  the  PMO,  the  IPT  committee,  or  the  independent  testing  and 
evaluator  groups,  a  serious  or  significant  vulnerability  will  remain  undetected  in  the 
fielded  system.  One  objective  of  a  vulnerability  assessment  strategy  is  to  minimize  the 
risk,  a  vulnerability  assessment  risk  (VA  risk),  that  a  material  vulnerability  (i.e.,  one  that 
could  result  in  catastrophic  loss  of  system,  failure  to  complete  mission,  and/ or 
personnel  casualties)  will  exist  undetected  in  a  fielded  weapon  system.  The  VA  risk 
could  be  described  as  a  three-fold  risk:  inherent  risk ,  control  risk,  and  detection  risk.' 

Inherent  risk  is  the  susceptibility  of  the  weapon  system  to  material  or  significant 
system  and  personnel  vulnerabilities.  Assessing  the  level  of  inherent  risk  associated 
with  a  system  requires  an  understanding  of  the  environment  in  which  the  system  must 
function,  the  expected  mission(s)  of  the  system  and  the  tasks  required  to  ensure  mission 
success,  the  threats  the  system  is  likely  to  encounter,  and  the  potential  mitigating  effects 
provided  by  the  tactics  and  doctrine  of  the  system  users.  Information  relevant  to  the 
vulnerabilities  of  prior  models  of  the  system  or  similar  models,  the  complexity  of  the 
design  of  the  system,  and  the  impact  of  technological  developments  on  the  system  serve 
as  input  to  the  assessment  of  inherent  risk. 

Control  risk  is  the  risk  that  a  material  or  significant  vulnerability  will  not  be 
prevented  or  detected  during  the  design  and  production  phases  of  the  system  under  the 
control  structure  of  the  PMO  program  of  survivability  or  vulnerability  assessment.  An 
accurate  assessment  of  the  level  of  control  risk  associated  with  a  weapon  system 


‘  Inherent,  control,  and  detection  risks  are  terms  borrowed  from  a  financial  auditing  context.  Although 
there  are  parallels  between  the  auditing  of  financial  statements  and  the  assessment  of  vulnerability  in 
weapon  systems,  the  terminology  as  used  in  this  report  is  not  strictly  analogous  to  the  terminology  in 
auditing  contexts. 
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requires  an  analysis  of  the  survivability  or  vulnerability  assessment  program  (e.g., 
design  and  engineering  analysis,  experimental  testing,  etc.)  conducted  by  the  PMO  and 
the  critical  data  voids  addressed  in  this  program  prior  to  full-scale  production. 

Detection  risk  is  the  risk  that  a  material  vulnerability  will  not  be  discovered  by 
analyses,  T&E,  or  other  activities  conducted  prior  to  fielding  of  the  system  by  sources 
independent  of  the  manufacturer/ contractor(s)  and  PMO.  Detection  risk  is  a  function 
of  the  nature,  timing,  and  extent  of  tests  (or  other  activities)  that  are  performed  with  the 
objective  of  assessing  system  vulnerability.  The  level  of  detection  risk  is  managed  to  a 
great  extent  by  the  independent  testers  and  evaluators  of  the  system. 

It  is  proposed  that  in  a  cost-effective  vulnerability  assessment  process,  independent 
testers  and  evaluators  would  (1)  determine  the  level  of  detectable  risk  they  are  willing 
to  accept  on  the  bases  of  their  assessment  of  the  levels  of  inherent  and  control  risks 
associated  with  the  system  and  (2)  design  the  nature  and  extent  of  their  analyses/ tests 
on  the  bases  of  those  assessments.  In  other  words,  higher  levels  of  detection  risk  may 
be  acceptable  to  a  tester /evaluator  planning  the  T&E  of  a  system  associated  with  low 
levels  of  inherent  and  control  risks.  Only  very  low  levels  of  detection  risk,  however, 
may  be  acceptable  to  testers/evaluators  assessing  the  vulnerabilities  of  a  system 
associated  with  very  high  levels  of  control  and  inherent  risk. 

5.1.3  Risks  of  Vulnerability  Assessment  and  FU  SL  LFT&E.  As  discussed  in  the 
previous  section,  one  objective  of  a  vulnerability  assessment  strategy  is  to  minimize  the 
risk  that  a  material  vulnerability  will  exist  undetected  in  a  fielded  system.  The  level  of 
detection  risk  that  is  acceptable  in  independent  tests  is  contingent  upon  the  assessed 
levels  of  inherent  and  control  risks;  the  level  of  detection  risk  associated  with  a  testing 
activity  is  determined  in  a  large  part  by  the  extent  and  nature  of  the  testing  activities 
conducted  by  independent  agencies. 

As  discussed  previously,  an  understanding  of  the  missions  of  the  system,  the  tasks 
required  to  complete  those  missions,  the  combat  environment,  the  threats  likely  to  be 
encountered,  and  the  tactics  and  doctrine  of  the  system  users  is  important  in 
determining  the  relationship  between  the  system's  capabilities  and  the  measures  of 
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mission  success  (O3  4  mapping).  The  data  required  for  establishing  the  minimum  level 
of  capabilities  needed  for  mission  completion  are  also  important  to  assessing  the  level  of 
inherent  risk  associated  with  the  system.  The  susceptibility  of  the  weapon  system  to 
material  vulnerabilities  (i.e.,  inherent  risk)  should  be  an  important  consideration  in  the 
design  of  the  control  structure  that  is  established  under  the  direction  of  the  PMO  with 
the  objective  of  preventing  and  detecting  vulnerabilities  in  the  system  produced. 

In  a  cost-effective  vulnerability  assessment  plan  that  includes  FU  SL  LFT&E, 
independent  testers  and  evaluators  might  be  expected  to  restrict  the  scope  of  the  FU  SL 
LFT  to  a  greater  extent  for  a  system  that  has  a  well-established  PMO-directed 
survivability  or  vulnerability  assessment  program  (i.ev  low  control  risk  assessed)  than 
for  a  system  with  similar  inherent  risks  but  a  less  mature  or  less  developed  control 
structure  in  place. 

5.2  Cost-Effective  FU  SL  LFT&E.  The  manufacturer/contractor(s)  of  the  system 
and  the  PMO,  as  well  as  independent  agencies,  participate  in  the  execution  of  an 
effective  assessment  strategy.  FU  SL  LFT&E  is  one  of  several  possible  activities 
conducted  by  an  independent  agency  that  may  be  selected  as  an  element  of  a 
vulnerability  assessment  plan.  The  methodology  presented  in  this  report  (i.e.,  section  4) 
would  support  the  inclusion  of  FU  SL  LFT  in  a  vulnerability  assessment  plan,  if  it 
compared  favorably  with  other  elements  on  the  basis  of  identified  costs  and  benefits 
(i.e.,  capacities  to  address  critical  data  voids)  and  uncertainties  associated  with  the 
identified  impacts.  For  certain  covered  weapon  systems,  however,  FU  SL  LFT&E  is  a 
congressionally  legislated  element  of  the  assessment  plan,  not  a  discretionary  activity  of 
the  assessment  process  [1],  For  that  reason,  a  closer  look  at  increasing  the  cost- 
effectiveness  ofFU  SL  LFT&E  is  important. 

Many  of  the  suggestions  to  increase  the  cost-effectiveness  of  FU  SL  LFT&E  that 
were  offered  by  interviewees  can  be  better  understood  in  the  context  of  the  objectives  of 
cost-effective  assessment,  the  risks  of  assessment,  and  the  activities  of  FU  SL  LFT&E. 
Generally,  suggestions  fell  into  four  broad  categories:  the  prioritization  of  data  voids, 
the  consideration  of  alternative  activities  for  FU  SL  LFT&E  and  alternative  designs  of 
the  FU  SL  LFT,  the  evaluation  of  the  control  structure  under  the  PMO  direction,  and  the 
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review  of  the  outcomes  of  prior  and  current  FU  SL  LFT&E  activities.  The  suggestions  of 
interviewees  are  summarized  in  the  following  paragraphs. 

5.2.1  Prioritization  of  Data  Voids.  Prioritize  data  voids  to  ensure  limited  funds  are 
spent  effectively  in  addressing  the  most  critical  data  voids.  Consider  the  benefits  (i.e., 
data  to  address  critical  data  voids)  expected  to  be  received  from  engineered  vs.  random 
shots,  as  well  as  benefits  from  each  additional  shot  planned  with  attention  given  to  the 
priorities  established  for  the  identified  data  voids.  Consider  alternative  activities  in 
conjunction  with  a  restriction  of  the  scope  of  FU  SL  LFT&E  to  effectively  and  efficiently 
address  critical  data  voids. 

5.2.2  Consideration  of  Alternative  Activities  and  Designs  of  FU  SL  LFT&E. 

Explore  alternative  activities  to  investigate  the  critical  data  voids  of  the  system  that  are 
planned  to  be  addressed  by  FU  SL  LFT&E.  Consider  substitution  of  alternative 
activities  if  less  costly  to  conduct  than  FU  SL  LFT&E  and/or  occur  earlier  in  the 
acquisition  process  than  FU  SL  LFT&E,  at  a  time  when  fixes  may  be  more  feasible 
and/or  more  economical. 

Consider  the  cost-effectiveness  of  the  design  of  FU  SL  LFT  (e.g.,  random  vs. 
engineered  shots,  number  of  shots,  etc.).  Balance  the  objectives  of  sampling  efficiency 
and  avoidance  of  bias  perceptions  by  employing  smart  testing  strategies  (e.g.,  random 
sampling  from  combat  distribution). 

Encourage  modelers  and  testers  to  integrate  efforts  in  the  development  and  W&A 
of  M&S  tools.  Facilitate  discussions  early  in  the  acquisition  process  among  analysts, 
manufacturer/ contractor (s),  PMO,  and  independent  testers  and  evaluators  to  increase 
time  available  to  consider,  plan,  and  conduct  alternative  activities  for  addressing  critical 
data  voids. 

5.2.3  Evaluation  of  Control  Structure.  Evaluate  the  control  structure  established  by 
the  PMO  to  monitor  system  survivability / vulnerability.  Encourage  early  involvement 
of  the  evaluator  in  assessment  process  to  ensure  minimal  funds  are  spent  on  redundant 
testing  and  analyses.  In  planning  the  scope  of  the  FU  SL  LFT&E,  give  careful 
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consideration  to  the  activities  managed  by  the  control  structure  to  prevent  and  detect 
vulnerabilities  throughout  the  acquisition  process.  An  evaluation  of  the  control 
structure  in  place  that  leads  to  an  assessment  of  low  control  risk  and  the  subsequent 
restriction  of  FU  SL  LFT  scope  could  serve  as  an  incentive  to  PMO/manufacturer/ 
contractor(s)  to  develop  and  maintain  an  active  agenda  for  assessment  through  all 
phases  of  acquisition. 

5.2.4  Review  ofFU  SL  LFT&E  Outcomes.  Compile  a  database  relevant  to  FU  SL 
LFT&E  outcomes  and  actions  (i.ev  system  retrofit,  redesign,  reengineering,  etc.)  taken 
on  the  bases  of  those  outcomes  for  all  FU  SL  LFT&E  conducted  (i.e.,  ex  post  analyses). 
Include  in  the  database  data  relevant  to  fixes  proposed  and  fixes  funded  for  the  current 
model,  as  well  as  alterations  implemented  in  later  models.  Identify  unsuspected 
vulnerabilities  detected  in  testing.  Require  the  costs  of  FU  SL  LFT&E  components  to  be 
recorded  and  reported  according  to  a  consistent  methodology. 

Outcome  data  obtained  ex  post  T&E  could  assist  in  identifying  the  types  of  data 
voids  that  are  addressed  most  effectively  in  FU  SL  LFT&E  and  the  areas  in  which 
unexpected  vulnerabilities  are  detected  most  often.  This  information  would  provide 
valuable  input  to  (1)  the  design  of  PMO  control  structures  for  the  prevention  and 
detection  of  vulnerabilities  in  system  design  and  production  and  (2)  the  planning  of 
future  FU  SL  LFTs  by  IPT  committees,  testers,  and  evaluators. 

For  example,  FU  SL  LFT&E  is  conducted  with  the  expectation  of  increasing 
evaluators7  understanding  of  the  relationship  between  the  system  interaction  with 
threats  likely  to  be  encountered  in  combat  and  the  damage  vectors  associated  with 
system  and  personnel  vulnerability  following  the  interaction  (i.e.,  01/2  mapping). 
Assume  a  review  of  the  outcomes  of  prior  FU  SL  LFTs  finds  that  a  large  number  of  the 
significant  unexpected  vulnerabilities  or  unknown  unknowns  uncovered  in  FU  SL  LFT&E 
can  be  categorized  as  data  relevant  to  the  02/3  mapping —  data  that  explain  the 
relationship  between  damage  vectors  and  the  measures  of  capability.  Because  023  data 
voids  can  be  addressed  more  efficiently  and  effectively  by  other  types  of  assessment 
activities  (e.g.,  engineering  analyses,  CDT),  this  finding  may  indicate  additional 
resources  are  needed  to  direct  assessment  activities  that  address  02/3  relationships. 
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Similar  to  outcome  data,  cost  data  gathered  ex  post  from  completed  FU  SL  LFT&E 
activities  are  relevant  to  the  planning  of  future  FU  SL  LFT&E.  Cost  data  identified  by 
the  components  of  FU  SL  LFT&E  facilitate  the  identification  of  value-added  and  non¬ 
value-added  components,  as  well  as  efficiencies  and  inefficiencies  in  the  T&E  process. 

5.2.5  Contributions  ofFU  SL  LFT&E  to  V/L  Assessment.  Although  many  offered 
suggestions  for  improving  the  cost-effectiveness  of  survivability /lethality  assessment  as 
a  whole  and  FU  SL  LFT&E  in  particular,  only  a  few  of  those  interviewed  for  this  report 
discounted  the  benefits  or  contributions  of  FU  SL  LFT&E.  Contributions  of  FU  SL 
LFT&E  cited  in  interviews,  as  well  as  published  reports  and  articles,  included  the 
capacity  of  FU  SL  LFT&E  to  identify  both  suspected  and  unknown  vulnerabilities  and 
lethalities;  provide  insights  into  potentially  significant  damage  mechanisms  and  the 
quantification  of  their  contribution  to  system  V/L;  supply  data  for  the  calibration  and 
validation  of  lethality  and  vulnerability  models;  provide  insights  into  BDAR  potential; 
and  contribute  to  the  development  of  battle  preparation  procedures,  and  operational 
doctrine  and  tactics.  The  importance  of  the  high  visibility  of  FU  SL  LFT&E  to  ensuring 
Congressional  funding  for  V/L  assessment  and  to  providing  incentive  for  PMOs  and 
contractors  to  consider  V/L  issues  over  all  phases  of  acquisition  cannot  be  minimized. 

In  addition,  the  intangible  benefits  received  from  T&E  conducted  by  an  independent 
agency  that  serves  as  validation  for  the  survivability/ vulnerability  programs  of 
PMO/contractor(s)  must  be  considered  in  a  weighing  of  the  impacts  of  FU  SL  LFT&E. 

5.3  Cost-Effective  Vulnerability  Assessment  and  Acquisition  Process.  In  this 
study,  the  FU  SL  LFT&E  of  three  separate  weapon  systems  were  reviewed  with  the 
purpose  of  identifying  the  activities  of  FU  SL  LFT&E,  the  impacts  of  those  activities,  and 
the  complexities  in  the  reporting  of  the  impacts.  Various  approaches  to  address  the 
complexities  uncovered  were  considered.  This  report  describes  the  current  FU  SL 
LFT&E  process,  including  its  role  in  a  program  of  vulnerability  assessment,  and  offers 
suggestions  for  improving  the  cost-effectiveness  of  vulnerability  assessment  strategies. 

In  planning  a  LFT&E  program,  the  LFIPT  brings  together  the  expertise  of  analysts, 
testers,  evaluators,  system  PMO  personnel,  system  users,  and  others  to  identify  and 
prioritize  data  voids  of  the  system  and  to  consider  alternative  data  sources  in  the  design 
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of  an  effective  LFT&E  strategy.  In  the  review  of  the  current  planning  process  for  FU  SL 
LFT&E  activities,  evidence  was  found  that  the  V/L  community  has  consistently 
attempted  to  implement  smarter  testing  strategies,  develop  ways  to  select  shots  that 
yield  more  useful  information,  and  build  on  the  lessons  of  prior  FU  SL  LFT&E 
conducted. 

It  is  suggested,  however,  that  more  effort  is  needed  in  developing  cost-effective 
strategies  for  improving  the  assessment  programs  of  new  systems  vs.  systems  in  the 
middle  of  the  acquisition  process  with  a  majority  of  budget  resources  already 
committed  to  programs.  Developing  flexible  alternative  assessment  programs  early  in 
the  acquisition  process  affords  maximum  opportunity  to  increase  the  efficiency  and  the 
effectiveness  of  the  assessment  of  the  system.  Linking  vulnerability  assessment 
programs  to  system  design  and  training  programs  promotes  cost  effectiveness  in  both 
the  gathering  and  the  use  of  data  required  by  multiple  decision-makers. 

In  conclusion,  it  is  important  to  consider  the  relationship  of  a  cost-effective 
vulnerability  assessment  strategy  to  a  cost-effective  acquisition  process.  An  effective 
vulnerability  assessment  strategy  includes  both  an  analysis  of  the  likely  combat 
scenarios  and  a  determination  of  the  links  between  the  measures  of  battlefield 
effectiveness  and  the  measures  of  capability  required  of  the  system  to  complete  its 
mission.  Understanding  the  mission  of  the  system  and  associated  tactical  utility  of  the 
system  user  in  likely  combat  scenarios  is  the  basis  for  understanding  how  users  rely  on 
the  weapon  system — the  foundation  for  vulnerability  assessment. 

A  mission-based  or  utility-based  strategy  of  system  acquisition  advocates  the 
establishment  of  weapon  system  performance  requirements  after  a  careful  analysis  of 
the  relationship  between  system  capability  measures  and  battlefield  utility  measures 
(i.e.,  O34  mapping)  [26].  The  relevant  set(s)  of  system  performance  requirements  then 
becomes  the  basis  for  weapon  system  design  and  engineering. 

The  information  relative  to  combat  scenarios  and  users'  tactics  and  doctrine 
gathered  for  acquisition  decisions  also  serves  as  valuable  input  to  the  supporting 
vulnerability  assessment  process.  In  the  assessment  process,  an  evaluation  is  made  of 
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the  extent  to  which  a  system,  interacting  with  threats  likely  to  be  encountered  in 
combat,  retains  lire  capabilities  that  were  determined  at  the  time  of  acquisition  to  be 
needed  for  mission  success.  The  operational  requirements  for  the  system,  incorporating 
the  levels  of  relevant  capabilities  needed  by  the  system  for  mission  success,  define  the 
minimum  levels  of  the  system's  capabilities  that  must  be  retained.  Therefore,  measures 
of  the  system's  capabilities  serve  not  only  as  the  basis  of  the  system  design  but  also  as 
the  core  of  vulnerability  assessment  strategies. 

The  importance  of  understanding  the  relationship  between  the  tasks  that  must  be 
completed  by  the  system  to  ensure  a  successful  mission  and  the  minimum  levels  of 
capabilities  needed  by  the  system  to  complete  those  tasks  is  emphasized  in  the 
assessment  methodology  proposed  in  this  report.  It  makes  sense  that  the  operational 
requirements  of  new  systems  incorporate  the  minimum  levels  of  capabilities  needed  to 
complete  mission  tasks  AND  that  the  objective  of  vulnerability  assessment  be  to  determine 
the  extent  to  which  the  system  retains  those  minimum  levels  of  capabilities  it  needs  to 
complete  its  mission(s)  when  it  interacts  with  those  threats  it  is  likely  to  encounter  in 
combat. 

Traditionally,  the  link  between  V/L  Taxonomy  Level  3  (i.e.,  measures  of  capability) 
and  Level  4  (i.e.,  measures  of  effectiveness)  metrics  has  not  been  considered  to  be  in  the 
purview  of  the  V/L  analyst.  The  methodology  proposed  in  this  report,  however, 
suggests  that  effectiveness  mapping  (i.e.,  03  4  mapping)  must  be  the  starting  point  not 
only  for  the  design  of  the  weapon  system  and  the  development  of  operational 
requirements  but  also  for  the  assessment  of  the  system's  vulnerability  in  likely  combat 
environments.* 

The  demands  of  the  acquisition  process  grow  with  the  increase  in  complex  systems 
with  advanced  technology  attempting  to  operate  in  changing  combat  environments. 
This  report  emphasizes  the  efficiencies  that  are  gained  by  the  mapping  of  the  tasks 
required  for  mission  success  to  mission-relevant  capabilities  and  the  sharing  of  these 


*  There  have  been  some  attempts  by  researchers  to  identify  the  tasks  that  must  be  completed  for  mission 
success  (i.e.,  mission-to-task  decomposition)  and  to  model  the  time-sequencing  of  task  activities  [42-44]. 
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data  between  decisions-makers  responsible  for  the  design  of  the  weapon  system,  the 
assessment  of  system  vulnerability,  the  training  of  system  users,  and  the  repair  and 
maintenance  of  the  system.  In  a  similar  fashion,  the  data  gathered  in  vulnerability 
assessments  to  evaluate  the  effect  of  system  damage  on  the  mission-relevant  measures 
of  capability  (02/3  mapping)  provide  insights  into  battlefield  damage  and  repair 
activities,  feed  the  operational  test  evaluation,  and  serve  as  inputs  to  programs  of 
training  simulators  [14, 15]. 

In  recent  years,  it  has  become  increasingly  more  evident  that  the  system  contractor, 
user,  tester,  evaluator,  and  related  analyses  groups  must  join  in  a  cooperative 
relationship  to  increase  the  effectiveness  and  efficiency  of  the  entire  acquisition  process. 
Evidence  suggests  this  cooperative  relationship  of  principal  players  may  lead  not  only 
to  increased  cost-effectiveness  in  the  acquisition  process  but  also  to  improved  readiness 
for  systems  in  combat. 
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AMSTATR 
J  CHAPIN 

WARREN  MI  48397-5000 

1  US  ARMY  INFO  SYS  ENGRG  CMD 
AMSEL  IE  TD 
FJENIA 

FT  HUACHUCA  AZ  85613-5300 

1  US  ARMY  SIM  TRNG  INST  CMD 

AMSTI CG 
M  MACEDONIA 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3726 

1  US  ARMY  TRADOC 

BATTLELAB  INTEGRATION 
TECH  B  CONCEPTS 
DIR  ATCD  B 

FT  MONROE  VA  23561-5000 

1  ARMY  TRADOC  ANL  CTR 
ATRCW 
MR  KEINTZ 
WSMRNM  88002-5502 

1  US  ARMY  RESEARCH  LAB 

AMSRLSL 

PLANS  AND  PGMS  MGR 
WSMR  NM  88002-5513 

1  US  ARMY  RESEARCH  LAB 

AMSRLSL  E 
WSMR  NM  88002-5513 

1  US  ARMY  RESEARCH  LAB 

AMSRLSL  EA 
R FLORES 

WSMR  NM  88002-5513 

1  US  ARMY  RESEARCH  LAB 

AMSRLSL  EM 
J  PALOMO 

WSMR  NM  88002-5513 

1  US  ARMY  RESEARCH  LAB 

AMSRLSL  El 
J  NOWAK 

FT  MONMOUTH  NJ  07703-5602 


NO.  OF 

COPIES  ORGANIZATION 

ABERDEEN  PROVING  GROUND 

1  US  ARMY  DEV  TEST  COM 
CSTE  DTC  TT  T 
APG  MD  21005-5055 

1  US  ARMY  EVALUATION  CENTER 
CSTE  AEC  MR  HUGHES 

4120  SUSQUEHANNA  AVE 
APG  MD  21005-3013 

4  US  ARMY  EVALUATION  CENTER 
CSTE  AEC  SV 
D  DELATTRE 
LFILLINGER 
S  GRILL 
R  LAUGHMAN 
4120  SUSQUEHANNA  AVE 
APG  MD  21005-3013 

2  US  ARMY  RESEARCH  LAB 
AMSRL  SL 

DR  WADE 
J  BEILFUSS 

4  US  ARMY  RESEARCH  LAB 
AMSRL  SLB 

MS  SMITH 
J  FRANZ 
M  VOGEL 
W  WINNER 

5  US  ARMY  RESEARCH  LAB 
AMSRL  SL  BA 

M  RITONDO 

5  US  ARMY  RESEARCH  LAB 
AMSRL  SLBD 
J  MORRISSEY 

21  US  ARMY  RESEARCH  LAB 
AMSRL  SL  BE 
D  BELY  (20  CPS) 

R  SANDMEYER 

5  US  ARMY  RESEARCH  LAB 
AMSRL  SL  BG 
A YOUNG 

5  US  ARMY  RESEARCH  LAB 
AMSRL  SL  BN 
D  FARENWALD 
APG  MD  21010-5423 
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NO.  OF 

COPIES  ORGANIZATION 

ABERDEEN  PROVING  GROUND  CONTINUED 

1  US  ARMY  RESEARCH  LAB 
AMSRL  SLE 
M STARKS 

1  US  ARMY  RESEARCH  LAB 
AMSRL  SLEA 
D  BAYLOR 

1  US  ARMY  RESEARCH  LAB 
AMSRL  SL  EM 
J  FEENEY 

1  US  ARMY  RESEARCH  LAB 
AMSRL  SLEC 
E  PANUSKA 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  SAIC 

6200  UPTOWN  BLVD 
SUITE  300 
C  E  ADOLPH 

ALBUQUERQUE  NM  87110 

1  DIRECTOR 

GEORGIA  INSTITUTE 
OF  TECHNOLOGY 
GTRI/ELSYSTEST  &  EVAL  RSCH 
&  EDUCATION  CTR 
DR  S  M  BLANKENSHIP 
ATLANTA  GA  30332-0840 

1  CHIEF 

CREW  STATION  R&D  BRANCH 
MANAGER  ADVANCED 
M&S  COMPLEX 
N  BUCHER 

AMES  RESEARCH  CTR  MS  243  3 
MOFFETT  FIELD,  CA  94035-1000 

1  DIR 

US  ARMY  MODEL  & 

SIMULATION  OFFICE 

DAMO  ZS  W  LUNCEFORD 

1 1 1 1  JEFFERSON  DAVIS  HIGHWAY 

CRYSTAL  GATEWAY  III 

SUITE  503  E 

ARLINGTON  VA  22202 

1  US  TACOM  ARDEC 
OR  ANALYST 
AMSTA  AR  FSS  J  C  CHU 
PICATINNY  ARSENAL  NJ 
07806-5000 

1  DIRECTOR 

CTR  FOR  PROFESSIONAL 
DEVELOPMENT  &  TNG 
UNTV  OF  TX  AT  AUSTIN 
DR  JG  DAVIS 
4030  2  W  BRAKER  LN 
AUSTIN  TX  78769-5329 


1  DIRECTOR 

SYSTEMS  ANALYSIS 
LOCKHEED  MARTIN 
ELECTRONICS  &  MISSILES 
RE  DOUGLAS 
5600  SAND  LAKE  RD  MP  21 1 
ORLANDO  FL  32819-8907 

1  DIRECTOR 

GEORGIA  TECH  RSCH  INSTITUTE 
ARLINGTON  RESEARCH  LAB 
WEEAGAR 

1700  N  MOORE  ST  SUTE  1910 
ARLINGTON  VA  22209 

1  AAAV  TECHNOLOGY  CTR 

DRPM  AAAV 
DWERDLEY 
991  ANNAPOLIS  WAY 
WOODBRIDGE  VA  22191-1215 

1  OPERATIONS  RSCH  ANALYST  USAF 
CBT  WPNS  SYS  ANLYS  DIV 
HQ  ACC  XRYS 
N  LEVANS 

204  ODD  VLVD,  STE  226 
LANGLEY  AFB  VA  23665-2777 

1  GEORGIA  TECH  RSCH  INSTITUTE 
GA  INSTITUTE  OF  TECHNOLOGY 
ELECTRONIC  SYSTEMS  LAB 
CONCEPTS  ANALYSIS  DIV 
J  R  GIBBONS 
ATLANTA  GA  30332-0800 

1  THE  MITRE  CORP 
SEN  M&S  ENGINEER 
P  A  GLASOW 

1820  DOLLEY  MADISON  BLVD  W626 
MCLEAN  VA  22102 

1  NEWPORT  NEWS  SHIPBUILDING 
JRGRIFFEN 
4101  WASHINGTON  AVE 
NEWPORT  NEWS  VA  23607-2770 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  APPLIED  RSCH  ASSOCIATES  INC 
W  HACKER 
4300  SAN  MATEO  NE 
SUITE  A  220 

ALBUQUERQUE  NM  871 10 

1  CF  HAWKINS 
CONSULTANT 
13140  LAKEHDLL  DR 
NOKESVILLE  VA  20181-3328 

1  ADUSA(OR) 

UNDER  SECY  OF  THE  ARMY 
DR  RG  HINKLE 
SAUS  OR 
RM  1E643 

102  ARMY  PENTAGON 
WASHINGTON  DC  20310-0102 

1  SR  SYSTEMS  ANALYST 
R  KANE 

1250  ACADEMY  PARK  LOOP 
SUITE  119 

COLORADO  SPRINGS  CO 
80910-3707 

1  CDR 

USA  TACOM 
AMSTA  UB  M  J  KING 
WARREN  MI  48397-5000 

1  JOINT  WARFARE  ANALYSIS  DEPT 
APPLIED  PHYSICS  LAB 
JOHNS  HOPKINS  UNIVERSITY 
WMKROSHL 
1 1 100  JOHNS  HOPKINS  RD 
LAUREL  MD  20723-6099 

1  LOCKHEED  MARTIN 

AERONAUTICAL  SYSTEMS 
WEKRUPP 
86  S  COBB  DR 
MARIETTA  GA  30063-0484 

1  PROGRAM  AREA  MNGR 

JOHNS  HOPKINS  UNIVERSITY 
APPLIED  LABORATORY 
DRRRLUMAN 
11100  JOHNS  HOPKINS  RD 
LAUREL  MD  20723 


1  OPERATIONS  RSCH  ANALYST 

PROGRAM  ANALYSIS  AND  EVAL 
DIRECTORATE 
MARCORSYSCOM  PAE 
D  G  MAYLUM 

2033  BARNETT  AVE  SUITE  315 
QUANTICO  VA  22134-5010 

1  US  ARMY  TACOM 
SFAEGCSSWBVQ 
M  S  NEIDELMAN 
WARREN  MI  48397-5000 

1  DR  R  NEW 

10959  DEBORAH  DR 
POTOMAC  MD  20854 

1  DEPUTY  DIRECTOR 
OSD/DOT&E/LFT&E 
RM  1C730A 
JFOBRYON 

1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  PARICIO  SPECIALTY  ENGRG  CO 
R  PARICIO 

16775  COUNTY  RD  362 
BUENA  VISTA  CO  81211 

1  ANALYST  OPERATIONS  RSCH 
KH  PILGRIM 
45  C  WILSON  RD 
WEST  POINT  NY  10996 

1  US  ARMY  TACOM 
PMGSIHTIDIV 
MRYZYI 

PM  SURVIVABILITY  SYSTEMS 
SFAEGCSSWGSIH 
WARREN  MI  48397-5000 

1  IDR  INC 

DIRECTOR  OF  BUS  DEVEL  &  SYS 

W  R  SCHILLING 

SUITE  515 

10560  MAIN  ST 

FAIRFAX  VA  22030 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  SCIENCE  ADVISOR  TO  THE  DIR 
OSD/DOT&E 
DR  E  A  SEGUE 
1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  APPLIED  RSCH  LABORATORIES 
THE  UNIV  OF  TX  AT  AUSTIN 
J  SHEEHAN 
AUSTIN  TX  78713-8029 

1  LOGICON 

DR  C  STANIEC 

2100  WASHINGTON  BLVD 

ARLINGTON  VA  22204-5706 

1  LOGICON  SYSCON 
E  SWARTZ 

2341  JEFFERSON  DAVIS  HWY 
SUITE  1138 

ARLINGTON  VA  22202 

3  INSTITUTE  FOR 

DEFENSE  ANALYSES 
DR  TONNESSEN 
DR  TURNER 
EUSANIO 

1801  N  BEAUREGARD  ST 
ALEXANDRIA  VA  223 11-1733 

3  SURTVCE  ENGINEERING  CO 
D  VAN  DUSEN 
A LAGRANGE 
E  EDWARDS 

1003  OLD  PHILADELPHIA  RD 
SUITE  103 

ABERDEEN  MD  21001-401 1 

1  USARMYTACOM 
SFAE  ASM  BV  CC 
J  P  VELLIKY 
WARREN  MI  38497-5000 

1  HEAD  WARFARE  PRGMS  T&E 

OFC  OF  CNO  OPNAV  N912C 
PRESIDENTIAL  TOWER  3D  FLR 
CPT  R  L  VIRGELIO  USN 
2000  NAVY  PENTAGON 
WASHINGTON  DC  20350-2000 


1  CHIEF  SCIENTIST  TECH  DIR 

AFOTEC  CN 
DR  ML  WILLIAMS 
KIRTLAND  AFB  NM  87117-5558 

1  EPVISCO 

3752  CAPULET  TERRACE 
SILVER  SPRING  MD  20906 

1  BOOZ  ALLEN  &  HAMILTON  INC 
LH  WOODS 
SUITE  131 

4141  COL  GLENN  HWY 
BEAVERCREEK  OH  45431 

1  HQ  SWC  AEA 
MAJ  S  BEERS 
730  IRWIN  AVE  SUITE  83 
SCHRIEVER  AFB  Cl  80912-7383 

1  HQDA 
ODCSOPS 

DAMO  ZD  V  BETTENCOURT 
400  ARMY  PENTAGON  RM  3A538 
WASHINGTON  DC  20310-0400 

1  US  ARMY  TEST  &  EVALUATION 
MANAGEMENT  AGENCY 
DACSTE 
DR  FOULKES 
PENTAGON  RM3C567 
WASHINGTON  DC  203 10 

1  RHELMUTH 

12  KIMBERLY  CIR 
BRUNSWICK  ME  040 11 

1  THE  MITRE  CORP 

MS  625  W150  M  LA  VINE 
1820  DOLLEY  MADISON  BLVD 
MCLEAN  VA  22102-3481 

1  US  ARMY  COMMUNICATIONS 
ELECTRONICS  CMD 
AMSEL  RD  DB  MAJ  LOCKHART 
BLDG  2700 

FT  MONMOUTH  NJ  07703 


NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  JOINT  ADVDIST  SIM 
JOINT  TEST  AND  EVAL 
TECHNICAL  ADVISOR  E  KECK 
1 1 104  MENAUL  BLVD  NE 
ALBUQUERQUE  NM  871 12 

4  UNIVERSITY  OF  PITTSBURGH 
KATZ  SCHOOL  OF  BUSINESS 
J  BIRNBERG 
JH  EVANS 
V  HOFFMAN 
D  MOSER 

PITTSBURGH  PA  15260 
20  CHAIR 

FRANKLIN  AND  MARSHALL 
COLLEGE 

DEPARTMENT  OF  BUS  ADMIN 
MK  NELSON 

LANCASTER  PA  17604-3003 

1  APPLIED  RSCH  ASSOCIATES  INC 
R  SCUNGIO 

219  W  BELAIR  AVE  SUITE  5 
ABERDEEN  MD  21001 

1  ASI  TEST  CONSULTANT 

G  BURGNER 
825  N  DOWNS  SUITE  C 
RIDGECREST  CA  93555 

1  BOOZ  ALLEN  &  HAMILTON  INC 
K  CROSSWAITE 
WPAFB  OH  45433-7542 

3  OSD/DOT&E/LFT&E 
T  JULIAN 
FMELLO 

R  WOJCIECHOWSKI 
RM  1C730 

1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  PM  ATACMS  BAT 

SFAE  MSL  AB  C  Q  T  R  CHELEKIS 
RSA  AL  35898-5650 


1  QUANTUM  RSCH  INTERN ATNL 

D  BRISTOL 
BEL  AIR  MD 

1  QUESTECH  INC 
M  EGLESTON 
901  N  STUART  ST 
SUITE  605 

ARLINGTON  VA  22203 

1  DIR  OPERATIONAL  TEST 
&  EVALUATION 
P COYLE 
RM3E318 

1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  DIR  OPERATIONAL  TEST 
&  EVALUATION 
HMAIR 

1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  US  ARMY  DEVELOPMENTAL 
TEST  COMMAND 
CSTE  DTC  TTT  N  HARRINGTON 
314  LONGS  CORNER  RD 
APG  MD  21005-5055 

1  US  ARMY  TEST  AND  EVALUATION 
COMMAND 
B  BARR 

PARK  CENTER  IV 
4501  FORT  AVE 
ALEXANDRIA  VA  22303 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRL  PBP  J  SCZEPANSKI 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  NAVAL  SURFACE  WARFARE  CTR 

CARDEROCK  DIVISION 
FFISCH 

BETHESDA  MD  20084-5000 
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NO.  OF 
COPIES 

1 

1 


1 

1 

1 

1 

1 

1 

1 


ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


NAVAL  SURFACE  WARFARE  CTR 
DAHLGREN  DIVISION 
G  24  T  WASMUND 
17320  DAHLGREN  RD 
DAHLGREN  VA  22448-5 100 

OFFICE  OF  NAVAL  RESEARCH 
TECH  ARA  MGR  SURF  AERO 
WEAPONRY  CODE  35 
D  SIEGEL 
800  N  QUINCY  ST 
ARLINGTON  VA  22217-5660 

USAF  PHILLIPS  LABORATORY 
PL  WS  W  BAKER 
3550  ABERDEEN  AVE  SE 
KIRKLAND  AFB  NM  87117-5576 

USAF  WRIGHT  LABORATORY 
WLFIVS 
M LENTZ 

WRIGHT  PAT  AFB  OH  45433-6553 

USAF  WRIGHT  LABORATORY 

1901  TENTH  ST 

RLAUZZE 

WRIGHT  PAT  AFB  OH  45433-7605 

USAFASCXRA 
BLDG  16 

2275D  ST  SUITE  10 
H  GRIFFIS 

WRIGHT  PAT  AFB  OH  45433-7227 

USAF  OPER  TEST  &  EVAL  CMD 
AFOCTE  CN 
M  WILLIAMS 
8500  GIBSON  BVLD  SE 
KIRKLAND  AFB  NM  87117-5558 

PEO  AIR  &  MSL  DEFENSE 
SFAE  AMD 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 

PEO  AEE  &  MSL  DEFENSE 
SFAE  AMD  ME 
POBOX  1500 

HUNTSVILLE  AL  35807-3801 


1  PEO  AIR  &  MSL  DEFENSE 

SFAE  AMD  THA 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 

1  PEO  AIR  &  MSL  DEFENSE 

SFAE  AMD  PA 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 

1  PEO  AVIATION 
SFAE  AV 
BLDG  5681 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AV  A  AH 
BLDG  5681 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AV  AAH  ATH 
BLDG  5681 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AV  AAH  LBA 
BLDG  5681 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AV  IR 
BLDG  5681 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AV  CH 
BLDG  5678 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AVCH  ICC 
BLDG  5681 
RSAAL  35898 

1  PEO  AVIATION 
SFAE  AV  RAH 
BLDG  5681 
RSAAL  35898 


NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS 
BLDG  171 

PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  W  AB 

WARREN  MI  48397-5000 

2  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  W  AB 

6501  11  MILE  RD 
WARREN  MI  48397-5000 

4  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAECGSSWBV 
WARREN  MI  48397-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  CR 
BLDG  171 

PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 

SFAE  GCSS  CR  A 
BLDG  171 

PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  CR  M 
BLDG  171 

PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 

SFAE  GCSS  JLW 
BLDG  151 

PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 

SFAE  GCSS  W  MTV 
WARREN  MI  48397-5000 


1  PEO  GRD  CMBT  &  SPT  SYSTEMS 

SFAE  GCSS  W  GSI 
BLDG  229 

WARREN  MI  48397-5000 

1  PRO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  W  MR 
MAIL  STOP  508 
WARREN  MI  48397-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 
AMSTA  TR  D/FSCS 

MAIL  STOP  275 
WARREN  MI  48397-5000 

2  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  ARMS 
PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 
SFAE  GCSS  TMA 
BLDG  354 

PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  GRD  CMBT  &  SPT  SYSTEMS 

SFAE  GCSS  TMA  SM 
PICATINNY  ARSENAL  NJ 
07806-5000 

1  PEO  TACTICAL  MISSILES 
SFAE  MSL 
BLDG  5250 
RSAAL  35898-8000 

1  PM  AIR  TO  GRD  MSL  SYSTEMS  OFC 

SFAE  MSL  HD 
RSA  AL  35898-5610 

1  PEO  TACTICAL  MISSILES 
SFAE  MSL  AB 
BLDG  5250 
RSAAL  35898-5650 

1  PM  ARMY  TACMS  BAT  PROJ  OFC 
SFAE  MSL  ABPB 
RSAAL  35898 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  PM  ARMY  TACMS  BAT  PROJ  OFC 

SFAE  MSL  AB  PI 
LTC  K  BROWINING 
RSA  AL  35898 

1  PM  ARMY  TACMS  BAT  PROJ  OFC 

SFAE  MSL  ABPP 
RSA  AL  35898 

1  PM  CLOSE  CMBT  ANT-ARMOR 
WPNS SYS  PROJ  OFC 
SFAE  MSL  CC 
RSA  AL  35898-5710 

1  PRODUCT  MANAGER 

ADVANCED  TOW  ACQUISITION 
SYSTEM 

SFAE  MSL  CC  AT 
RSA  AL  35898-8000 

1  PMO 

SFAE  MSL  AM 
BLDG  3651 
RSA  AL  35898-5720 

1  PEO  TACTICAL  MISSILES 
MLRS 

SFAE  MSL  ML 
BLDG  5250  RM  B200 
RSA  AL  35898-5700 

1  PEO  TACTICAL  MISSILES 

HIGH  MOBILITY  ARTILLERY 
ROCKET  SYSTEM 
SFAE  MSL  ML  HM 
RSA  AL  35898-5710 

1  PEO  TACTICAL  MISSILES 

MLRS  IMPROVED  LAUNCHER 
MECHANICAL  SYSTEM 
SFAE  MSL  ML  IL 
RSA  AL  35898-5700 

1  PEO  TACTICA  MISSILES 
MLRS  PRECISION  GUIDED 
MUNITIONS 
SFAE  MSL  ML  PGM 
RSA  AL  35898-5700 


1  US  ARMY  SPECIAL  OPS  CMD 

TECHNLGY  APPLICATION  PROG  OFC 
AMSAMRDAAZT 
BLDG  401  LEE  BLVD 
FT  EUSTIS  VA  23604-5577 

1  DSA  US  ARMY  AVN  &  MSL  CMD 

AMSAMDSA 
RSA  AL  35898-5000 

1  US  ARMY  AVN  &  MSL  CMD 

AMSAM  DSA  MP 
MULTI  PURPOSE  INDIVIDUAL 
MUNITION/SHORT  RANGE 
ASSAULT  WPN 
RSA  AL  35898 

1  US  ARMY  AMCOM 

AMSAM  DSA  AS  ASH 
SCOUT/ATTACK 
RSA  AL  35898 

1  US  ARMY  AMCOM 

SHORT  RANGE  AIR  DEFENSE 
AMSAM  DSA  SH 
RSA  AL  35898 

1  US  ARMY  AMCOM 
UTILITY  HELICOPTER 
AMSAM  DSA  UH 
RSA  AL  35898 

1  US  ARMY  AMCOM 

UNMANNED  GROUND  VEH 
AMSAM  DSA  UG 
RSA  AL  35898 

1  US  ARMY  SOLDIER  &  BIOLOGICAL 
CHEMICAL  COMMAND 
LAND  WARRIOR 
10401  TOTTEN  RD  SUITE  121 
FT  BELVOIR  VA  22060-5852 

1  US  ARMY  TACOM 
AMSTALC 

WARREN  MI  48397-5000 

1  US  ARMY  TACOM 
AMSTALC  AH 
WARREN  MI  48397-5000 


NO.  OF 

COPIES  ORGANIZATION 


1  USARMYTACOM 
AMSTA  LC  AL 
WARREN  MI  48397-5000 

1  US  ARMY  TACOM 

AMSTA  LC  AM 
PICATINNY  ARSENAL  NJ 
07806-5000 

1  MARINE  CORPS 

INTELLIGENCE  ACTIVITY 
DRCPM  AAAV  T  M  LEAF 
991  ANNAPOLIS  WAY 
WOODBRIDGE  VA  22191-1215 

1  PM  ABRAMS  TANK  SYSTEM 

SFAE  GCSS  W  AB  QT  P  LEITHEISER 
WARREN  MI  48397-5000 

ABERDEEN  PROVING  GROUND 

22  DIR  USAMSAA 
EATZINGER 
P  BEAVERS 
W  BROOKS  JR 
COLBUCHA 
P  H  DEITZ  (10  CPS) 

CFOX 
W  HE  APES 
W  NIEMEYER 
B PARKS 
TRUTH 
D  J  SHATTER 
R  THOMPSON 
A  WONG 
APG  MD  21005 


Form  Approved 
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